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Introduction 


Imagine  that  a  drunkard  comes  out  of  a  pub  at  closing  time.  Every  five  seconds 
(say)  he  staggers  one  metre  up  the  road  or  one  metre  down  the  road,  with  equal 
probability.  The  random  process  used  to  model  his  erratic  wanderings  up  and 
down  the  street  is  a  simple  example  of  a  random  walk;  this  example  is  known  as 
the  drunkard’s  walk. 

Now  suppose  that  the  police  station  is  20  metres  down  the  road  from  the  pub,  and 
that  if  the  drunkard  happens  to  arrive  outside,  the  station  sergeant  hauls  him 
inside  and  locks  him  up  for  the  night.  The  police  station  acts  as  an  absorbing 
barrier  for  the  random  walk— it  is  a  barrier  as  the  drunkard  cannot  pass  the 
station  and  it  is  absorbing  since  once  there  he  cannot  leave.  This  modified  process 
is  called  a  random  walk  with  one  absorbing  barrier.  If,  in  addition,  he  lives  50 
metres  up  the  street  on  the  other  side  of  the  pub  from  the  police  station  and,  once 
home,  his  wife  takes  him  in  off  the  doorstep,  then  his  home  acts  as  a  second 
absorbing  barrier;  and  this  modified  process  is  called  a  random  walk  with  two 
absorbing  barriers. 

What  is  the  probability  that  the  drunkard  will  be  locked  up  for  the  night?  Or  that 
he  will  reach  home?  Is  one  or  other  of  these  certain,  or  can  he  wander  up  and 
down  the  road  indefinitely?  These  are  the  types  of  question  you  should  be  able  to 
answer  after  studying  Section  2.  In  that  section,  a  classical  gambling  problem, 
called  the  gambler's  ruin ,  is  described;  this  provides  another  example  of  a  situation 
that  can  be  modelled  by  a  random  walk  with  two  absorbing  barriers. 

But  first,  in  Section  1,  a  random  walk  is  defined  and  it  is  shown  that  a  random 
walk  is  an  example  of  a  more  general  class  of  processes  called  Markov  chains 
(these  will  be  studied  in  Unit  6).  Random  walks  with  absorbing  barriers  and 
random  walks  with  reflecting  barriers  will  also  be  discussed  briefly.  An  example  of 
a  reflecting  barrier  is  given  by  a  further  modification  of  the  drunkard’s  walk.  If, 
instead  of  locking  up  the  drunkard,  the  sergeant  sets  him  on  his  way  towards  his 
home,  then  the  police  station  acts  as  a  reflecting  barrier.  Random  walks  with 
reflecting  barriers  are  discussed  in  Section  5.  The  rest  of  the  unit  (Sections  3  and  4) 
is  concerned  with  unrestricted  random  walks — i.e.  random  walks  without  barriers. 

The  sections  of  this  unit  are  not  independent  of  each  other;  the  later  sections  draw 
on  ideas  and  results  from  the  earlier  ones,  so  you  should  study  the  sections  in 
order.  In  addition,  many  of  the  ideas  in  this  unit  are  discussed  in  more  detail  in 
Unit  6;  this  applies  particularly  to  some  of  the  material  in  Sections  4  and  5.  So  if 
you  find  any  of  the  material  in  these  sections  difficult,  in  Unit  6  you  will  have 
another  opportunity  to  sort  out  the  ideas.  Also,  in  that  unit,  many  examples  of 
random  walks  are  used  to  illustrate  ideas  about  Markov  chains,  so  you  should 
complete  your  study  of  this  unit  before  going  on  to  the  next. 

There  are  two  short  sequences  of  the  video-cassette  associated  with  this  unit; 

Band  C  illustrates  ideas  from  Subsection  3.2  and  Band  D  is  linked  to  Subsection 
5.1.  These  are  best  watched  after  studying  the  relevant  subsections;  each  one 
requires  about  15  minutes  of  your  time. 

There  is  also  an  audio-tape  session  in  this  unit.  On  the  audio-tape,  which  forms 
part  of  Section  2,  the  gambler’s  ruin  problem  is  discussed  and  a  solution  is 
described  which  makes  use  of  the  Theorem  of  Total  Probability.  You  will  also 
need  to  solve  a  second-order  recurrence  relation;  a  method  of  solving  such 
recurrence  relations  is  given  in  the  Handbook  so,  if  you  are  not  used  to  solving 
them,  you  may  wish  to  study  this  before  listening  to  the  tape.  In  any  case,  you 
may  find  it  helpful  to  revise  this  material. 

In  Section  3  the  Central  Limit  Theorem  is  used  to  estimate  various  probabilities 
associated  with  random  walks;  so  in  this  section  you  will  need  to  use  the  table  of 
probabilities  from  the  normal  distribution  in  Neave.  In  Section  4  generating 
functions  are  used  to  obtain  some  further  results;  you  will  need  to  be  familiar  with 
just  a  few  basic  ideas  about  probability  generating  functions.  The  Binomial 
Theorem  is  also  used  in  this  section;  all  the  information  you  need  is  in  the 
Handbook. 


1  What  is  a  random  walk? 


1.1  Simple  random  walks 

We  begin  by  taking  another  look  at  the  Bernoulli  process;  this  simple  example  will 
be  used  to  introduce  some  terminology  and  another  type  of  process — the  random 
walk. 

Example  1.1  The  Bernoulli  process 

You  will  recall  that  a  Bernoulli  process  is  a  sequence  of  Bernoulli  trials  in  which 

(i)  the  trials  are  independent; 

(ii)  the  probability  of  success,  p ,  remains  constant  from  trial  to  trial. 

Denoting  a  successful  trial  by  1  and  a  failure  by  0,  the  outcome  of  the  nth  trial  is 
an  observation  on  a  Bernoulli  random  variable  Y„,  where  P(Y„  =  1)  =  p  and 
P(Yn  =  0)  =  q  =  1  —  p.  The  process  (Y„;  n  =  1,2,3, ...}  is  particularly  simple:  for 
each  n,  the  only  possible  values  of  Yn  are  0  and  1.  So  the  set  of  all  the  values  that 
can  be  taken  by  the  random  variables  is  {0, 1};  this  set  is  the  state  space  of  the 
random  process.  If  Y„  =  0,  then  the  process  is  in  state  0  at  time  n;  and  if  Yn  =  1, 
then  the  process  is  in  state  1.  □ 

Example  1.2 

A  related  sequence  of  random  variables  is  defined  as  follows.  Let  X„  represent  the 
number  of  successes  accumulated  by  time  n  in  a  sequence  of  Bernoulli  trials.  So, 
for  each  n>  1, 

xh-y1  +  y2  +  ...  +  yh, 

where  Yu  Y2, ...,  Yn  are  independent  Bernoulli  random  variables  with  parameter  p\ 
hence,  Xn  ~  B{n,  p)  and 

P(X„  =  x)  =  j  p  V',  X  =  0, 1, 2 . n. 

Table  1.1  below  shows  the  first  twelve  observations  in  a  realization  of  the  process 
(Y„;  n  =  1,2,  ...}  and  the  associated  process  {Xn;  n  =  1,2, ...},  where  p  =  0.4. 


Table  1.1 


n 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

Yn 

0 

1 

1 

0 

1 

0 

0 

1 

1 

0 

1 

1 

xn 

0 

1 

2 

2 

3 

3 

3 

4 

5 

5 

6 

7 

If,  in  addition,  we  set  X0  =  0,  then  the  process  {Xn\  n  =  0, 1,2, ...}  is  in  state  0 
before  the  first  trial.  Thus,  the  process  begins  in  state  0  and,  after  each  trial,  either 
remains  where  it  is  (if  the  trial  is  unsuccessful)  or  moves  up  to  the  next  state  (if  the 
trial  results  in  a  success);  in  this  way,  the  process  {Xn}  moves  from  0  through  the 
positive  integers.  In  the  realization  in  Table  1.1,  the  process  began  in  state  0  and 
moved  through  the  states  1,2,  ...,  6,  reaching  state  7  by  time  12;  further 
observations  would  eventually  result  in  state  8  being  visited,  then  9,  and  so  on. 
Thus,  the  state  space  for  the  process  {X„;  n  =  0, 1,2, ...}  is  the  set  of  non-negative 
integers. 

In  one  application  of  this  process,  Xn  is  the  position  at  time  n  of  a  particle  moving 
along  a  line.  This  application  will  be  described,  and  used  to  introduce  some 
standard  terminology. 

Suppose  a  particle  is  moving  along  a  line.  It  starts  at  the  origin  and,  at  regular 
intervals,  either  moves  or  remains  where  it  is  according  to  the  following  rules:  with 
probability  p  it  moves  one  unit  to  the  right  and  with  probability  q  =  1  —  p  it 
remains  where  it  is;  this  is  shown  in  Figure  1.1. 
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Figure  1.1  The  process  {Af„;  «  =  0, 1,2, ...} 


At  each  time  n  (n  =  0, 1,2,  ...)  a  ‘change’  in  the  state  of  the  process  occurs;  these 
‘changes’  are  called  transitions.  For  this  process,  two  types  of  transition  are 
possible:  from  state  x  to  state  x  +  1  (with  probability  p )  and  from  state  x  to  state 
x  (with  probability  q  =  1  —  p);  these  are  shown  in  Figure  1.2.  If  the  process  is  in 
the  same  state  x  at  time  n  (n  =  1, 2,  . . .)  as  at  time  n  —  1,  we  still  say  that  a 
transition  has  occurred  at  time  n:  the  term  ‘transition’  does  not  necessarily  imply 
movement  to  a  different  state. 

This  process  is  an  example  of  a  random  walk;  it  is  one  in  which  steps  to  the  left 
and  steps  greater  than  one  are  impossible.  This  random  walk  is  completely  defined 
by  the  initial  condition  X0  =  0,  which  fixes  the  starting  point  of  the  walk  at  the 
origin,  and  by  p  and  q,  the  transition  probabilities: 

P(X„  =  x  +  l\Xn.1  =  x)  =  p, 

P(Xn  =  x  |  Xn.  x  =  x)  =  q  =  1  -  p. 

Now  Xn,  the  position  of  the  particle  at  time  n,  depends  only  on  Ar„_1,  the  position 
at  time  n  —  1,  and  Yn,  the  change  in  position  at  the  nth  step;  and,  for  n  >  1, 

Xn  =  Xn-1  +  Yn-  P(Yn  =  1)  =  p,  P(Yn  =  0)  =  q. 

Moreover,  for  each  n,  Yn  is  independent  of  A'„_1 ;  that  is,  the  distribution  of  the  nth 
step  is  independent  of  the  particle’s  position  immediately  before  that  step.  The 
essential  feature  of  this  process  (and  of  any  other  random  walks)  is  that  the  steps, 
or  increments,  Yx,  Y2, are  independent  identically  distributed  random 
variables.  □ 
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Figure  1.2  Transitions  for 
{Xn\  n  =  0, 1, 2,  . . .} 


This  random  walk,  in  which  steps  to  the  left  are  impossible,  suggests  the  possibility 
of  more  general  random  walks  in  which  steps  in  both  directions  are  allowed,  or  in 
which  steps  of  two  or  more  units  may  occur.  The  situation  in  the  next  example 
naturally  gives  rise  to  a  random  walk  in  which  the  particle  can  make  unit  steps 
either  to  the  left  or  to  the  right. 

Example  1.3  Coin-tossing 

A  fair  coin  is  tossed.  If  it  lands  head-side  up,  then  you  win  lp;  and  if  it  lands  tail- 
side  up,  then  you  lose  lp.  Let  X„  represent  the  number  of  heads  obtained  minus 
the  number  of  tails  obtained  in  the  first  n  tosses;  and  let  the  initial  state  of  the 
game  be  given  by  X0  =  0.  If  X„  is  positive  at  time  n,  then  you  are  ahead  in  the 
game;  if  X„  is  zero,  then  you  are  even;  and  if  X„  is  negative,  then  you  have  less 
money  than  you  started  with. 


Figure  13  Three  realizations  of  {Xn\  n  =  0, 1,2, ...}:  number  of  heads  minus  number  of  tails 

Figure  1.3  shows  the  first  twelve  observations  in  each  of  three  realizations  of  the 
process  { Xn ;  n  =  0, 1,2,  ...}.  In  the  first  realization  there  are  more  heads  than  tails, 
so  Xx2  is  positive;  in  the  second  there  are  equal  numbers  of  heads  and  tails 
( Xx2  =  0);  and  in  the  third  there  are  fewer  heads  than  tails  ( Xx2  <  0). 


Notice  that  Xn  also  represents  your  net  gain  (or  loss)  after  n  tosses;  so,  if  Zn 
represents  the  amount  you  win  on  the  outcome  of  the  nth  toss  then,  for  n  >  1, 

XH  =  Zl+Z2  +  ...  +  Z„; 

and,  for  i  >  1,  P(Z*  =  1)  =  \  and  P{Zi  =  —  1)  =  Also,  your  net  gain  after  n 
tosses  is  equal  to  your  net  gain  after  n  —  1  tosses  plus  the  amount  you  win  on  the 
nth  toss;  that  is, 

Xn  =  Xn-i  +  z„. 

Another  way  of  interpreting  the  process  {X„;  n  =  0, 1,2,  ...}  is  to  think  of  Xn  as 
representing  the  position  on  a  line  of  a  particle  which  starts  at  the  origin  and,  at 
regular  intervals,  takes  a  unit  step  either  to  the  left  or  to  the  right  with  equal 
probability;  this  is  illustrated  in  Figure  1.4. 


The  process  { Xn ;  n  =  0, 1,2,  ...}  is  another  example  of  a  random  walk;  the 
increments,  Z1,Z2,  ...,  are  independent  identically  distributed  random  variables; 
so  that,  for  each  n  >  1,  Z„  is  independent  of  Xn_x—  that  is,  the  nth  step  is 
independent  of  the  particle’s  position  immediately  before  that  step.  □ 

Question  1.1  What  is  the  state  space  of  the  process  in  Example  1.3?  □ 

For  the  random  walk  in  Example  1.2,  the  possible  transitions  are  from  x  to  x  +  1 
with  probability  p  and  from  x  to  x  with  probability  q  =  1  —  p.  And  in 
Example  1.3,  the  possible  transitions  are  from  x  to  x  +  1  and  from  x  to  x  -  1, 
each  with  probability  These  are  both  examples  of  random  walks  of  a  type 
generally  referred  to  as  simple  random  walks;  at  each  step  of  such  a  random  walk, 
the  particle  moves  one  unit  to  the  left,  remains  where  it  is,  or  moves  one  unit  to 
the  right,  and  no  other  steps  are  possible.  The  use  of  the  term  ‘simple’  to  describe 
such  a  random  walk  is  not  universal;  some  authors  reserve  it  to  describe  the 
particularly  ‘simple’  random  walk  in  Example  1.3.  However,  in  this  course,  a 
simple  random  walk  is  a  random  walk  in  which,  at  each  step,  the  particle  either 
remains  where  it  is  or  moves  one  unit  to  the  left  or  right;  so  that,  from  n  >  1,  the 
distribution  of  Z„,  the  nth  step,  is  of  the  form 

P(Zn  =  1  )  =  p,  P(Zn  =  -  1  )  =  q,  P(Zn  =  0)  =  r,  (1.1) 

where  p  +  q  +  r  =  1.  (The  symbols  p,  q  and  r  will  always  have  this  interpretation 
in  the  context  of  simple  random  walks.) 

The  requirement  which  must  be  met  in  general  for  a  process  to  be  a  random  walk 
is  that  the  steps  are  independent  and  identically  distributed.  For  a  simple  random 
walk  the  size  of  each  step  is  either  zero  or  one  unit;  for  a  general  random  walk 
this  restriction  is  relaxed,  and  steps  of  any  size  are  allowed.  These  requirements  are 
contained  in  the  following  definition. 


A  random  process  {Xn;  n  =  0, 1,2,  ...}  is  a  random  walk  if,  for  n  >  1, 

Xn  =  Xn.1+Zn, 

where  {Z,},  i  =  1,2, ...,  is  a  sequence  of  independent  identically  distributed 
random  variables.  If,  in  addition,  the  only  possible  values  of  each  Z,  are  - 1, 
0  and  1,  then  the  process  is  a  simple  random  walk. 
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Question  1.2  Suppose  that  you  toss  two  fair  coins  together;  for  two  heads  you 
win  lp;  for  two  tails  you  lose  lp;  and  for  one  head  you  neither  win  nor  lose.  Let 
Z„  represent  your  gain  on  the  nth  toss  and  let  Xn  represent  your  net  gain  after  n 
tosses;  then  {Xn\  n  =  0, 1,2, ...}  is  a  simple  random  walk. 

(i)  What  are  the  values  of  p,  q  and  r? 

(ii)  Use  the  thirty-first  row  of  the  table  of  random  digits  on  page  42  of  Neave 
(beginning  59998)  to  simulate  the  first  ten  observations  in  a  realization  of  the 
process.  □ 

1.2  Random  walks  with  barriers 

Example  1.4 

Suppose  that,  in  the  coin-tossing  game  of  Example  1.3,  you  are  prepared  to  lose 
only  50  p;  so  you  have  decided  that  you  will  stop  playing  if  at  any  stage  you  have 
50  p  less  than  you  started  with.  Can  a  random  walk  be  used  to  model  this 
modified  game? 

As  long  as  x,  the  amount  gained,  is  greater  than  —  50,  we  have 
P(Xn  =  x  +  1  |X„_,  =  x)  =  P(Zn  =  1)  =  i 

and 

P(Xn  =  x-\\Xn  =  x)  =  P{Z„  =  -1)  =  h 
as  before.  But  when  x  =  —  50,  this  is  no  longer  true;  for  if  at  any  time  your  losses 
reach  50  p,  you  will  stop  playing,  so  your  net  loss  will  remain  50  p  thereafter;  that 
is, 

P(Xn=  -50\Xn_1=  -50)=  1. 

In  other  words,  once  the  state  —50  is  entered  it  cannot  be  left;  this  is  illustrated  in 
Figure  1.5.  A  state  which  once  entered  cannot  be  left  is  called  an  absorbing  state. 

In  this  example,  the  state  —  50  also  prevents  the  random  walk  from  moving  below 
—  50;  it  is  called  an  absorbing  barrier.  So  the  process  {Xn;  n  =  0, 1,2, ...}  may  be 
interpreted  as  describing  the  motion  of  a  particle  which  moves  as  if  executing  a 
simple  random  walk  as  long  as  its  position  is  above  the  point  —50;  if  it  visits  that 
point,  it  remains  there.  This  process  is  called  a  simple  random  walk  with  an 
absorbing  barrier  at  —  50.  □ 

Question  1.3  What  is  the  state  space  of  the  process  in  Example  1.4?  □ 

Question  1.4  Suppose  that  in  Example  1.4  your  opponent  is  prepared  to  lose  no 
more  than  20  p  and  has  decided  that  he  will  stop  playing  if  his  losses  ever  reach 
this  figure.  The  resulting  process  {X,,;  n  =  0, 1,2,  ...}  is  a  random  walk  with  two 
absorbing  barriers. 

(i)  Where  is  the  second  absorbing  barrier? 

(ii)  What  is  the  state  space  of  this  random  walk? 

(iii)  Draw  a  diagram  to  illustrate  this  random  walk.  □ 

There  are  various  questions  which  we  can  ask  about  a  random  walk  with 
absorbing  barriers.  What  is  the  probability  of  absorption  at  each  barrier?  That  is, 
for  each  absorbing  barrier,  what  is  the  probability  that  from  any  given  starting 
position  the  particle  will  eventually  enter  that  barrier  state?  Is  absorption  certain 
to  occur  eventually?  Or  can  the  particle  move  backwards  and  forwards  between, 
but  not  reaching,  the  barriers  indefinitely?  And  if  absorption  is  certain,  when  is  it 
likely  to  occur?  All  these  problems  will  be  investigated  in  Section  2  when  we  look 
at  a  classical  problem  known  as  the  gambler's  ruin. 

An  absorbing  barrier  is  not  the  only  type  of  barrier;  another  is  described  in  the 
following  example. 


Figure  1.5  Absorbing  barrier  at  -  50 


Not  all  absorbing  states  are 
absorbing  barriers:  for  example,  for 
a  random  walk  with  steps  of  size 
greater  than  one,  an  absorbing  state 
may  be  stepped  over. 
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Example  1.5 


Consider  a  particle  which  moves  on  a  line  between  the  points  a  and  b,  where  a 
and  b  are  integers  and  b  >  a.  It  starts  at  some  point  strictly  between  a  and  b  and, 
as  long  as  X„,  its  position  after  n  steps,  is  strictly  between  these  points  then,  at  the 
next  step,  it  moves  either  one  unit  to  the  left  with  probability  $  or  one  unit  to  the 
right  with  probability  f ;  that  is,  for  a  <  x  <  b, 

P(Xn  =  x+  l|X„_1=x)  =  i, 

P(Xn  =  x-  l|Z„_1  =  x)  =  i. 

This  is  illustrated  in  Figure  1.6. 

However,  if  at  any  time  n  >  1  the  particle  enters  one  of  the  states  a,  b,  then  at  the 
next  step  it  is  returned  to  the  adjacent  state  with  probability  1 ;  that  is, 

P(X„  =  a  +  l\XH-i  =  a)  =  1, 

P(Xn  =  b-l\Xn_1  =  b)=l. 

This  is  illustrated  in  Figure  1.7. 


(i)  The  reflecting  barrier  at  a  (ii)  The  reflecting  barrier  at  b 

Figure  1.7 


The  state  space  is  {xeZ:  a  <  x  <  b}. 
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Figure  1.6  Transitions  for  x 
between  a  and  b 


If  the  particle  reaches  a  or  b  at  time  n  >  1,  then  it  is  ‘reflected’  back  to  its  previous 
position;  the  states  a  and  b  are  reflecting  barriers.  The  process  {X„;  n  =  0, 1,2,  ...} 
is  called  a  simple  random  walk  with  reflecting  barriers  at  a  and  b.  □ 


There  is  another  type  of  barrier  which  can  also  be  described  as  a  reflecting  barrier 
(sometimes  called  a  ‘partially  reflecting  barrier’);  this  is  introduced  in  the  next 
example.  This  type  of  reflecting  barrier  will  be  discussed  in  Section  5. 


Example  1.6 

Suppose  that  the  particle  in  Example  1.5  is  moving  as  before  except  that  when  it 
enters  one  of  the  barrier  states  a,  b  there  is  a  chance  that  it  remains  there  instead 
of  being  returned  immediately  to  the  neighbouring  state;  for  example,  the 
transition  probabilities  from  a,b  could  be  as  follows: 

P(Xn  =  a+  l\Xn_l=a)  =  $,  P(Xn  =  a\Xn-1=a)  =  $, 

P(Xn  =  b-i\Xn.l  =  b)  =  },  P(X„  =  b\Xn-i  =  b)  =  $. 

This  is  illustrated  in  Figure  1.8. 


Figure  1.8  Transition  probabilities  at  the  reflecting  barriers 


One  way  of  interpreting  this  type  of  behaviour  at  a  barrier  state  is  the  following. 
Suppose  that  the  particle  is  in  state  a  and  that  there  is  a  probability  of  j  of 
moving  one  unit  to  the  right  and  ^  of  moving  one  unit  to  the  left;  imagine  that 
there  is  a  mirror  or  a  wall  at  a  —  1  and  that  if  the  particle  tries  to  move  to  state 
a  -  1  it  is  immediately  reflected  back  to  a;  so  it  never  actually  reaches  a  -  1  but  is 
‘held’  at  a.  Similarly,  one  can  imagine  another  mirror  placed  at  b  +  1.  For  this 
reason  this  process  is  also  referred  to  as  a  random  walk  with  reflecting 
barriers.  □ 

The  questions  we  might  wish  to  ask  about  a  random  walk  with  reflecting  barriers 
are  not  the  same  as  for  a  random  walk  with  absorbing  barriers;  it  no  longer  makes 
sense  to  talk  about  the  probability  of  absorption  or  the  time  to  absorption. 

Instead,  we  might  be  interested  in,  for  example,  the  proportion  of  time  spent  in  the 
different  states;  this  is  one  of  the  problems  which  will  be  discussed  in  Section  5. 


Some  authors  describe  this  as  a 
random  walk  with  reflecting  barriers 
at  a  —  1  and  6+1,  others  place  the 
barriers  at  a  —  \  and  b  +  exactly 
where  the  barriers  are  placed  is  not 
important — it  is  the  idea  that 
matters. 


1.3  Random  walks  and  Markov  chains 


Finally  in  this  section,  two  particular  properties  of  a  random  walk  will  be 
discussed  briefly  as  these  are  also  possessed  by  a  much  wider  class  of  random 
processes.  The  first  of  these  is  called  the  Markov  property.  For  any  random  walk,  if 
the  distribution  of  Z„  is  known,  then  the  distribution  of  the  position  of  the  particle 
after  n  steps,  X„,  is  determined  by  its  position  after  n  —  1  steps;  its  position  at  any 
earlier  time  is  irrelevant  if  its  position  after  n  -  1  steps  is  known.  That  is,  the 
distribution  of  Xn  is  determined  by  the  value  of  Xn_x  \  and,  if  X„^1  is  known,  then 
any  information  about  Xn_2,Xn_3,  ...,  X0  has  no  effect  on  the  conditional  (on 
^n-i)  distribution  of  Xn.  For  example,  the  distribution  of  X2  is  determined  by  the 
value  of  Xlt  and  the  distribution  of  X3  is  determined  by  the  value  of  X2;  the  value 
of  X0  provides  no  information  about  X2  that  is  not  given  by  the  value  of  Xx ;  and 
X0  and  Xx  provide  no  information  about  X3  that  is  not  given  by  the  value  of  X2. 
That  is,  only  the  most  recent  piece  of  information  is  relevant  to  the  future 
behaviour  of  the  process. 

In  general,  a  stochastic  process  in  discrete  time,  {Xn;  n  =  0, 1,2, ...},  is  said  to  have 
the  Markov  property  if  the  distribution  of  each  random  variable,  Xn  (n  >  1),  is 
determined  by  the  value  of  the  preceding  one,  Xn_1;  so  that,  once  the  value  of 
Xn-x  is  known,  the  distribution  of  Xn  is  not  affected  by  any  additional  information 
about  the  history  of  the  process.  A  stochastic  process  in  discrete  time  which  has 
the  Markov  property  is  called  a  Markov  chain;  so  every  random  walk  is  a  Markov 
chain.  However,  not  every  Markov  chain  is  a  random  walk,  as  can  be  seen  from 
the  following  simple  example. 

Example  1.7 

In  a  breakfast-cereal  promotion,  a  plastic  animal  is  given  away  with  each  packet; 
there  are  three  different  animals,  and  each  packet  is  as  likely  to  contain  any  one  of 
them  as  any  other.  Let  Xn  represent  the  number  of  different  animals  that  a 
particular  family  has  obtained  after  buying  n  packets  of  cereal.  Clearly,  the 
distribution  of  Xn  depends  only  on  the  value  taken  by  A-*.!,  so  the  process 
{Xn;  n  =  0, 1, 2, . . .}  is  a  Markov  chain. 

Question  1.5 

(i)  What  is  the  state  space  of  the  process  {Xn;  n  =  0, 1,2, ...}  of  Example  1.7? 

(ii)  Find  the  distribution  of  X„  given 

(a)  _ j  =  0,  (b)  Xn_x  =  1,  (c)  An_1  =  2,  (d)  Xm-x  =  3. 

(iii)  Is  state  3  reflecting,  absorbing  or  neither?  □ 

The  transition  probabilities  found  in  Solution  1.5(ii)  are  illustrated  in  Figure  1.9. 


i  * 


Figure  1.9  Transition  probabilities 
for  {X„;  u  =  0,1,2, ...} 


As  you  will  see  in  Unit  6,  the  style  of  the  figure  is  that  for  Markov  chains  and  it  is 
used  here  to  emphasize  that  this  Markov  chain  is  not  a  random  walk,  as  is  now 
shown.  In  the  definition  of ‘random  walk’,  the  steps  Z1,Z2, ...  are  independent 
identically  distributed  random  variables.  In  particular,  this  means  that  the 
transition  probabilities  for  a  random  walk  are  independent  of  current  position;  for 
example, 

P(Xn  =  a  +  1  |  An_  j  —  a)  =  P(Xn  =  b  +  1 1  An_j  =  b).  Each  of  these  probabilities  is  equal 

But  in  this  Markov  chain  we  have,  for  instance,  to  p^n  = 

P(Xn  =  2|*„_1  =  l)  =  f  and  P(Xn  =  3\Xn_x  =2)  =  *; 
so  this  Markov  chain  is  not  a  random  walk.  □ 
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The  second  property  of  a  random  walk  which  is  also  possessed  by  some  other 
types  of  random  process  is  that  the  transition  probabilities  P(Xn  =  y  \X„_1  =  x), 
n  =  1,2,  are  independent  of  n.  (This  follows  because  Zi,Z2  are  independent 
and  identically  distributed.)  The  transition  probabilities  are  said  to  be  stationary 
over  time.  So,  for  example,  the  probability  that  the  next  step  is  a  unit  step  to  the 
right,  from  xtox+  1,  is  the  same  whether  the  particle  is  in  state  x  after  10  steps, 
or  after  100  steps,  or  after  1000  steps.  This  property  also  holds  for  the  Markov 
chain  in  Example  1.7;  the  transition  probabilities  found  in  Solution  1.5  do  not 
change  with  time.  The  process  {Xn;  n  =  0, 1,2,  ...}  in  that  example  is  a  Markov 
chain  with  stationary  transition  probabilities.  All  the  Markov  chains  that  you  will 
meet  in  this  unit  and  the  next  have  stationary  transition  probabilities;  so  most  of 
the  time,  for  simplicity,  such  processes  are  referred  to  as  Markov  chains  and  the 
phrase  ‘with  stationary  transition  probabilities’  is  omitted.  This  point  will  be 
discussed  in  Unit  6,  when  a  formal  definition  of  a  Markov  chain  will  be  given. 

The  class  of  processes  known  as  Markov  chains  includes  processes  of  several  types. 
Firstly,  it  includes  processes  in  which  the  random  variables  Xn  are  independent, 
such  as  the  Bernoulli  process;  it  also  includes  random  walks,  which  are  processes 
with  independent  increments;  it  includes  simple  branching  processes;  and  it 
includes  many  other  processes— Example  1.7  is  but  one.  You  will  meet  many  more 
examples  of  Markov  chains  in  the  next  unit;  but  now  we  return  to  the  main  theme 
of  this  unit — random  walks. 


2  The  gambler’s  ruin 


Whatever  your  personal  feelings  about  gambling,  probability  theory  can  increase 
your  understanding  of  games  of  chance;  the  rules  of  probability  can  be  applied  to 
many  gambling  situations  in  order  to  calculate  the  probability  of  winning  or 
losing.  In  this  section,  we  shall  look  at  just  one  example— the  classical  problem  of 
the  gambler’s  ruin.  Although  simple,  it  provides  an  impressive  illustration  of  the 
results  of  continued  gambling;  we  study  it  here  as  it  is  a  famous  and  interesting 
example  of  a  random  walk  with  absorbing  barriers. 

Consider  a  gambler,  whom  we  shall  call  Gary,  who  plays  a  series  of  games  against 
an  opponent;  after  each  game  the  loser  pays  the  winner  £1.  Suppose  that,  in  each 
game,  he  wins  or  loses  £1  with  probabilities  p  and  q  (q  =  1  -  p),  respectively,  and 
that  he  continues  to  play  as  long  as  both  he  and  his  opponent  have  money  to 
gamble.  What  is  the  probability  that  Gary  eventually  loses  all  his  money?  We 
should  expect  this  probability  to  depend  not  only  on  p,  his  probability  of  winning 
each  game,  but  also  on  his  and  his  opponent’s  capital  at  the  start  of  the  match; 
intuitively,  it  would  appear  that  he  will  have  more  chance  of  winning  the  match  if 
he  begins  with  £20  and  his  opponent  with  £1  than  if  their  initial  fortunes  are 
reversed.  Another  question  we  might  ask  is  whether  one  or  other  of  the  gamblers 
is  certain  to  be  ruined  eventually  or  whether  it  is  possible  for  the  match  to 
continue  indefinitely.  And,  if  the  match  is  certain  to  end,  how  long  is  it  likely  to 
last?  These  are  some  of  the  problems  that  we  shall  investigate  in  this  section. 

Until  recently,  the  gambler’s  ruin  problem  has  generally  been  attributed  to 
Huygens;  in  his  treatise  De  Ratiociniis  in  Aleae  Ludo,  published  in  1657,  he  posed 
a  problem  which  is  equivalent  to  the  situation  of  fair  bets  (p  =  q  =  i).  A  solution 
of  the  more  general  problem,  for  any  given  initial  fortunes  and  any  given  value  of 
p,  was  later  given  by  James  Bernoulli  in  his  work  Ars  Conjectandi,  published  in 
1713,  eight  years  after  his  death.  However,  the  problem  goes  back  at  least  to 
Pascal;  in  1656  he  posed  to  Fermat  a  problem  which  is  a  special  case  of  the 
gambler’s  ruin.  It  is  clear  that  they  both  solved  the  problem,  although  no  record 
exists  of  their  methods.  That  Huygens  heard  of  the  problem  from  Pascal  was 
revealed  only  with  the  publication  of  Huygens’  correspondence  at  the  end  of  the 
last  century.  However,  this  was  after  Todhunter  published  his  History  of  the 
Mathematical  Theory  of  Probability  in  1865,  and  this  work  continues  to  this  day  to 
be  the  major  secondary  source  of  information  on  the  early  history  of  probability; 
so  Pascal’s  knowledge  of  the  problem  has  not  been  widely  recognized. 
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In  the  audio-tape  session  the  probability  of  ruin  will  be  calculated  when  the  games 
are  fair  and  when  one  player  has  an  advantage  over  the  other  (p  #  q);  we  shall 
also  look  at  the  situation  of  gambling  against  an  opponent  with  large  resources 
(such  as  a  casino).  But  first  we  shall  show  how  the  gambler’s  ruin  can  be  modelled 
by  a  simple  random  walk  with  absorbing  barriers. 


2.1  The  gambler’s  ruin  as  a  random  walk 

Suppose  that  Gary  begins  with  £j  and  his  opponent  has  £(a  -  j).  So  between  them 
they  have  £a,  and  they  continue  to  gamble  until  one  or  other  of  them  is  ruined. 

Let  Xn  be  Gary’s  fortune  after  the  nth  game;  he  begins  with  £j,  so  X0  =  j.  Let  Z, 
be  the  amount  he  wins  in  the  ith  game,  which  is  independent  of  the  amount  he 
wins  in  any  other  game;  then  each  Z;  takes  the  values  1  and  - 1  with  probabilities 
p  and  q  respectively;  that  is, 

P(Zt  =  1  )  =  p,  P(Z{  =  - 1)  =  q  =  1  -  p. 

Thus  Zj,Z2,  ...  is  a  sequence  of  independent  identically  distributed  variates. 

Gary’s  fortune  after  the  nth  game  is  the  sum  of  his  fortune  after  the  (n  -  l)th  game 
(provided  this  is  greater  than  zero  and  less  than  a)  and  the  amount  he  wins  in  the 
nth  game: 

Xn  =  Xn_  i  +  Z„,  where  0  <  Xn_  t  <  a. 

If  either  Xn-x  =  0  (in  which  case  Gary  is  ruined)  or  Xn_1  =  a  (his  opponent  is 
ruined),  then  the  match  is  over  and  Xn  —  Xn_1.  These  probabilities  are 
summarized  in  Figure  2.1. 


Figure  2.1  Transition  probabilities  for  the  gambler’s  ruin 


Thus,  the  process  {Xn;  n  =  0, 1,2,  ...},  which  has  state  space  (0, 1,2, ...,  a},  is  a 
random  walk  with  absorbing  barriers  at  0  and  a.  (The  probability  that  Gary  is 
eventually  ruined  is  just  the  probability  that  the  random  walk  is  absorbed  at  the 
barrier  0.)  Rather  than  talking  about  the  state  space  of  the  process,  it  is  natural  to 
describe  the  process  {Xn;  n  =  0, 1,2,  . . .}  as  a  random  walk  ‘on  the  integers 
0,1,2, ...,  a\ 


2.2  The  probability  of  ruin 


Let  qj  be  the  probability  that  Gary  is  ruined  if  he  begins  with  £j  (and  his  opponent 
with  £(a  -  j )).  If  he  begins  with  no  money,  then  he  cannot  place  any  bets  and  thus 
cannot  win  any  money;  so  q0  is  equal  to  1.  And  if  he  begins  with  £a,  then  he 
cannot  be  ruined — he  will  keep  all  his  money  as  his  opponent  has  none  to  gamble 
with;  so  qa  is  0.  But  what  is  the  value  of  qs  for  0  <  j  <  al 

The  method  that  will  be  used  to  solve  this  problem  involves  an  application  of  the 
Theorem  of  Total  Probability  and  the  solution  of  a  second-order  recurrence 
relation.  You  may  find  it  helpful  to  revise  these  briefly  before  listening  to  the  tape. 
The  information  you  will  need  about  recurrence  relations  can  be  found  in  the 
Handbook;  for  easy  reference,  the  Theorem  of  Total  Probability  is  stated  below. 

Theorem  of  Total  Probability  For  any  event  A, 

P(A)=  £  P(A\El)P{El), 

i=  1 

where  EX,E2, En  are  mutually  exclusive  and  exhaustive  events.  For  n  =  2  this 
reduces  to 

P(A)  =  P(A  |  E)  P(E)  +  P(A  |  E)  P(E); 
this  is  the  form  in  which  we  shall  apply  the  theorem. 

You  will  find  it  useful  to  have  paper  to  write  on  as  you  work  through  the 
problems  in  this  audio-tape  session. 


^T)  The  classical  gambler’s  ruin 

Conditions 

After  each  game  the  loser  pays  the  winner  £1 . 

For  each  game,  the  probability  that  Gary  wins  is  p. 

Gary  and  his  opponent  have  £a  between  them. 

Problem 

Find  the  probability  qj  that  Gary  is  ruined  if  he  begins  with  £j. 
Method 

I  Apply  the  Theorem  of  Total  Probability : 

P(A)  =  P(A  |E)  P(E)  +  P(A|E)  P(E). 

II  Solve  the  recurrence  relation  obtained. 


[^)  Solution 

I  Apply  the  Theorem  of  Total  Probability 

Rj  is  the  event  that  Gary  is  eventually  ruined  if  he  begins  with  £j. 
W  is  the  event  that  Gary  wins  the  first  bet. 

For  0  <j  <  a, 

P(Rj)  =  P(Rj|W)P(W)  +  P(Rj|w)P(W), 

t  t  t 

p  <\- 

P(Rj|W)  =  P(RJ+1)  =  qJ  +  I> 

PfRjIWJ  =  P(Rj_,J  =  q,-,  .  £ 

So  <\)  =  pqJ+i +  qqj-i.  j=i,2,...,a-i. 

Boundary  conditions :  q0  =  I,  qa  =  0. 


A  second -order 
homogeneous 
recurrence  s 
)  relation  \ 


^3)  II  Solve  the  recurrence  relation 

Step  f:  Try  gk  =  Cxk  and  obtain  the  auxiliary  equation: 

CxJ  =  p  CxJ  +  1  +  q  Cx}~] . 

So  the  auxiliary  equation  is  x  =  px2  +  q. 

Step Z:  Solve  the  auxiliary  equation: 

px2  -  x  +  q  =  0 

px2-  (p  +q)  x+  q  =  0  /p  +  q  =  1 

(px  -  g)(x  -  1)  =  0  ^ 


x  =  —  or  x  =  1 . 
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z)  steP3  :  Write  down  the  general  solution  of  the  recurrence  relation. 

Cose  I  Distinct  roots:  -3-^1,  i.e.  p  f  q. 

/  a  \J 


The  general  solution  is  =  A  +  B  (^p-j 
CaseZ  Equal  roots:  3-  =  1,  i.e.  p  =  q  =  ~ 

The  general  solution  is  qj  =  A  +  Bj . 

5^  Step  4:  Find  the  particular  solution  which  satisfies  the  boundary  conditions 

r  ncn  1  n  .  —  A  R  /  .T  \ 


q  a  o 


Case  1  q j  =  A  +  B  ^ 
q0  =  1  => 


So  A  = 


,  B  = 


and  qj  = 


CaseZ  qj  =  A  +  Bj 


q0  =  i 


So  A 


ga  0 
and  qj 


Gambling  against  a  casino 

Problem  Find  Gary’s  probability  of  rum  if  he  is  playing  against  a 
casino  with  unlimited  resources  and  he  begins  with  £j. 

Method  Find  qj  as  a  -►  oo. 

Solution 

j  fq/p)J  -  (q/p)a 

—  p#tl  ’’ ‘  ,  - (,/p)- 


(a)  p 

>q 

As  a  - 

*>, 

(b)  p 

<q 

As  a  - 

oo, 

*  = 

Case  Z 

p=q 

1 

2 

As  a  - 

>  00, 

3 


5A)  Case 


I  =  A  +  B 


%  =  0 


(  q\° 

0  =  A  +  B 


iMl! 


l-(q/p)a 


„  -  Kq/p)J-(q/p)a 


l-(q/p)a 


^6A)  Case 


0  =  A  +  ba 


The  gambler  5  ruin  (Conditions  in  Frame  1) 
The  probability  that  Gary  is  ruined  when  he 
begins  with  £j  is 

(g/p)J-  (q/p)a 

,J  I  -  fq/p)'  P/1 

1  "  1  P  '  1  *  I 


(a)  p  >q  As  a  oo, 


(b)  p  <q  As  a 


Case  2  p  =  q  =  j  As  a 


(p/q)fl 


Gambling  against  a  casino 

When  playing  against  a  casino  with 
unlimited  resources,  the  probability  of  rum 
when  beginning  with  £j  is 

f 1  pm 


p>q. 
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Question  2.1  Gary’s  opponent  begins  the  match  with  £(a  -  j )  and  has  probability 
q  of  winning  and  p  of  losing  each  bet. 

(i)  Use  the  results  in  Frame  5A  to  write  down  expressions  for  pj,  the  probability 
that  Gary  wins  the  match  (and  his  opponent  is  ruined),  (a)  when  p  =  q, 

(b)  when  p  #  q. 

(ii)  Hence,  show  that  the  match  is  certain  to  end  eventually.  □ 

Question  2.2  Gary  makes  bets  of  £1  each  on  the  tosses  of  a  fair  coin;  he  decides, 
in  advance,  to  stop  when  his  winnings  reach  £20  or  his  losses  reach  £10. 

(i)  State  an  equivalent  gambler’s  ruin  problem. 

(ii)  Find  the  probability  that  Gary  loses. 

(iii)  What  is  his  expected  loss?  □ 

Question  2.3  Gary  plays  a  match  in  which  he  has  probability  0.4  of  winning  any 
individual  bet.  He  begins  with  £10  and  his  opponent  initially  has  £90. 

(i)  Find  the  probability  that  Gary  is  ruined  if  each  bet  is  £1. 

(ii)  Now  suppose  that  they  stake  £5  on  each  bet.  What  is  the  probability  that 
Gary  is  ruined? 

(iii)  Find  the  probability  that  Gary  is  ruined  if  he  stakes  £10  on  each  bet. 

(iv)  What  size  bets  would  you  recommend  him  to  make?  □ 

2.3  The  duration  of  the  match 

Since  the  match  described  in  the  gambler’s  ruin  is  certain  to  end  eventually  in  the 
ruin  of  one  or  other  player,  one  question  of  interest  is  ‘How  long  will  the  match 
last?’.  One  way  of  answering  this  would  be  to  find  the  probability  distribution  of 
the  duration  and  hence  the  expected  duration  of  the  match.  However,  this  involves 
some  complicated  algebra,  so  we  shall  be  content  to  find  only  the  expected 
duration;  and  we  shall  do  this  by  a  much  simpler  method,  similar  to  that  used  to 
find  the  probability  of  ruin. 

We  shall  assume  that  the  duration  of  the  match,  when  Gary  begins  with  £j  (and  £1  bets  are  assumed. 

his  opponent  with  £(a  —  j)),  has  a  finite  expectation,  Dj.  To  find  Dj  we  shall  use  a 

result  for  expectations  from  Unit  2;  this  states  that,  for  random  variables  X  and  Y,  Unit  2,  Restrft~3:5' 

E(X)  =  Ey[E(X  |  Y)].  (2.1)  r|vvvvu^ 

If  we  let  X  be  the  duration  of  the  match  when  Gary  begins  with  £j,  then 
E(X)  =  Dj ;  and  if  Y  is  a  random  variable  representing  Gary’s  winnings  on  the  first 
bet,  which  takes  the  value  1  when  he  wins  the  first  bet  and  - 1  when  he  loses  the 
first  bet,  then 

P(Y=l)  =  p  and  P(Y=—l)  =  q. 

Applying  Result  (2.1),  we  have 
E(X)  =  Ey\_E(X  |  Y)] 

=  ££(*!  r=.y)P(y=.)0 

y 

=  E(X\Y=l)p  +  E(X\Y=  -1  )q. 

Now  E(X  |  Y=  1)  is  the  expected  duration  of  the  match  given  that  the  first  bet  is 
won  by  Gary.  But,  if  Gary  wins  the  first  bet,  the  match  continues  as  if  his  initial 
fortune  was  j  +  1,  with  expected  duration  Dj+1  after  the  first  bet,  so 
E(X  |  Y=  1)  =  1  +  Dj+1. 

Similarly,  if  Gary  loses  the  first  bet  ( Y=  - 1),  then  the  match  continues  as  if  his 
initial  fortune  was  j  —  1,  so 

E(X  |  Y  =  -1)  =  1  +Dj.l. 


So,  when  0  <  j  <  a,  the  expected  duration  (Dj  =  E(Xj)  satisfies  the  recurrence 
relation 

Dj  =  {  1  +  Dj+i)p  +  (1  +  Dj_i)q 
or 

Dj  =  pDj+1+qDj.l  +  1.  (2.2) 

When  j  =  0  or  j  =  a,  the  match  does  not  begin  so  its  expected  duration  is  zero. 

Thus  the  boundary  conditions  are 

Do  =  0,  Da  =  0.  (2.3) 

The  only  difference  between  the  second-order  recurrence  relation  (2.2)  and  the  one 
in  Frame  2  for  the  probability  of  ruin  is  the  appearance  of  the  constant  term  1, 
which  makes  it  non-homogeneous.  In  the  next  two  questions,  you  are  asked  to  find 
the  solution  first  when  one  player  has  an  advantage  and  then  when  the  bets  are  fair. 

Question  2.4 

(i)  For  the  gambler’s  ruin  problem  with  p  ^  q,  show  that  a  particular  solution  of 
the  recurrence  relation  (2.2)  is  given  by 


and  deduce  that  the  general  solution  is  given  by 

Dj  =  -J—  +  A  +  b(^1 

q-p  \pj 

where  A  and  B  are  constants. 

(ii)  Find  the  values  of  A  and  B  for  which  the  expression  for  Dj  in  part  (i)  gives  the 
solution  satisfying  the  boundary  conditions  (2.3).  □ 

Question  2.5 

(i)  For  the  gambler’s  ruin  problem  with  p  =  q  =  show  that  a  particular  solution 
of  the  recurrence  relation  (2.2)  is  given  by  Dj  =  —j2,  and  deduce  that  the 
general  solution  is  given  by  Dj  =  -j2  +  A  +  Bj,  where  A  and  B  are  constants. 

(ii)  Find  the  values  of  A  and  B  for  which  the  expression  for  Dj  in  part  (i)  gives  the 

solution  satisfying  the  boundary  conditions  (2.3).  □ 

The  last  two  questions  show  that  the  expected  duration  of  the  match  in  the 
classical  gambler’s  ruin  problem  is  given  by 


D  j  a  (I~(q/PY\ 

J  q-p  q-p\l-(q/p)a)’ 

when  p  ¥=  q. 

(2.4) 

dj  =j{a  -j). 

when  p  =  q  = 

(2.5) 

In  fact,  the  duration  of  the  match  is  likely  to  be  considerably  longer  than  might  be 
suggested  from  intuition  alone.  Try  the  following  question;  you  may  find  the 
results  surprising. 


Question  2.6 

(i)  Two  players,  initially  with  £500  each,  place  £1  bets  on  the  outcomes  of  tosses 
of  a  fair  coin  until  one  or  other  of  them  is  ruined.  What  is  the  expected 
duration  of  the  match? 

(ii)  If,  instead  of  having  £500  each,  one  player  has  £1  and  the  other  begins  with 
£1000,  what  is  the  expected  duration  of  the  match?  □ 


In  the  audio-tape  session,  we  found  the  probability  of  ruin  for  a  gambler  playing 
against  a  casino  with  unlimited  resources,  by  taking  the  limit  as  a  -»  oo  of  the 
probability  of  ruin  given  in  Frame  5 A.  When  p  is  greater  than  q,  there  is  a  positive 
probability  that  the  match  will  continue  indefinitely,  so,  in  this  case,  it  does  not 
make  sense  to  talk  about  its  expected  duration.  However,  when  p  is  less  than  q, 
the  expected  duration  can  be  found  by  taking  the  limit  as  a  -+  oo  in  the  expression 
(2.4)  for  Dj,  giving 


For  p  <  q. 


as 


a 


oo. 
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q-p 


When  p  =  q  =  the  expression  (2.5)  for  D}  increases  indefinitely  as  a  increases,  so 
the  expected  duration  is  infinite.  So,  although  ultimate  ruin  is  certain,  it  may  be  a 
long  time  before  it  happens. 


3  The  unrestricted  random  walk 


In  the  gambler’s  ruin  problem  the  fortune  of  a  player  is  modelled  by  a  random 
walk  with  two  absorbing  barriers  (at  0  and  a),  or,  in  the  case  of  gambling  against  a 
casino,  one  absorbing  barrier  (at  0).  As  long  as  the  particle  is  situated  at  one  of  the 
states  1,2,  ...,  a  —  1  at  time  i  —  1,  the  next  step,  Zh  has  the  probability 
distribution 

P(Zi  =  1)  =  P,  P(Zt  =  - 1)  =  q  =  1  -  p;  (3.1) 

and  Xh  the  position  of  the  particle  at  time  i,  is  given  by  =  X,-_  j  +  Z,.  But  if  the 
particle  enters  one  of  the  barrier  states,  then  it  remains  there. 

Now  suppose  that  the  absorbing  barriers  are  removed;  then  Equations  (3.1)  give 
the  distribution  of  the  z'th  step,  Z,,  independently  of  the  particle’s  position  before 
this  step  so  that,  for  all  i  >  1  and  for  any  value  of  Xi_1,  we  have 

Xi  =  Xl-l  +  Zt. 

We  think  of  the  particle  as  moving  along  a  line  and,  at  each  stage,  taking  a  unit 
step  to  the  right  (+ 1)  with  probability  p  or  to  the  left  (—1)  with  probability  q.  So, 
if  the  particle  is  initially  at  the  origin  (X0  =  0),  its  position  after  n  steps  is 

X„  =  Zx  +  Z2  +  ...  +  Zn. 

There  are  various  questions  that  can  be  asked  about  this  random  walk.  For 
example,  what  is  the  probability  of  returning  to  the  origin  at  some  future  time  n? 

Is  eventual  return  certain?  Or  is  it  possible  for  the  particle  to  wander  away  and 
never  return?  What  is  the  distribution  of  the  particle’s  position  after  n  steps?  How 
far  away  from  its  starting  point  is  it  likely  to  be?  And  can  we  find  a  range  of 
positions  within  which  it  is  almost  certain  to  be? 

In  this  section  we  shall  study  each  of  these  problems  in  turn.  We  begin  by  finding 
the  probability  that  the  particle  will  eventually  return  to  the  origin. 


3.1  Return  to  the  origin 


The  method  by  which  we  shall  find  the  probability  that  a  particle  starting  at  the 
origin  eventually  returns  there  makes  use  of  one  of  the  results  of  the  previous 
section.  And  once  again  we  begin  by  applying  the  Theorem  of  Total  Probability. 


Let  R  be  the  event  that  a  return  to  the  origin  occurs  eventually.  We  wish  to  find 
P(R);  this  probability  will  be  denoted  by  /.  Let  A  be  the  event  that  the  first  step  is 
to  the  right,  so  that  if  A  occurs  then  after  the  first  step  the  particle  is  at  1,  and  if  A 
occurs  then  it  is  at  —  1 ;  and  P{A)  =  p  and  P(A)  =  q  =  1  —  p.  By  the  Theorem  of 
Total  Probability, 


/  =  P(R)  =  P(p  |  A)  P{A)  +  P(R  |  A)  P(A).  (3.2) 

The  values  of  the  two  probabilities  P(R  \  A)  and  P(R  \  A)  can  be  written  down  using 
the  result  for  gambling  against  a  casino,  which  is  given  in  Frame  6 A  of  the  audio- 
tape  session.  This  states  that,  when  a  gambler  is  playing  against  a  casino  which 
has  unlimited  resources,  his  probability  of  ruin  if  he  begins  with  £j  and  stakes  £1 
on  each  bet  is  qj,  where 


p<q 
p>  q 


(3.3) 


and  p  is  the  probability  that  he  wins  an  individual  bet  and  q  =  1  —  p  is  the 
probability  that  he  loses  a  bet.  In  random  walk  terminology,  this  states  that  a 


particle  which  starts  at  j  and,  at  each  step,  moves  one  unit  to  the  right  with 
probability  p  or  one  unit  to  the  left  with  probability  q  =  1  —  p,  has  probability  qj 
of  ever  visiting  the  origin.  But  P(R  \  A )  is  the  probability  of  ever  returning  to  the 
origin  when  the  first  step  is  to  the  right;  and  this  is  the  same  as  the  probability, 
starting  in  state  1,  of  ever  visiting  the  origin;  that  is,  P(R  \A)  —  qx.  Hence,  we  have 
1  p  <  q 

P(R\A)  =  \q  (3.4) 

p  P>q. 

Similarly,  P(R  \  A)  is  the  probability  of  ever  moving  from  —  1  to  0;  and  we  can  use 

the  gambler’s  ruin  interpretation  to  write  down  this  probability  too.  It  is  equal  to 

the  probability  of  ever  moving  from  1  to  0  in  a  simple  random  walk  in  which  the 
probability  of  a  positive  step  is  q  (instead  of  p).  And  this  is  equal  to  the  probability 
of  ruin  of  a  player  who  begins  with  £1  and  has  probability  q  of  winning  a  bet. 
Thus,  replacing  p  by  q  and  vice-versa  in  Formula  (3.4),  we  obtain 
1  p  >  q 

P(R\A)  =  \p  (3.5) 

-  P<q. 

When  q  <  p,  substituting  Formulas  (3.4)  and  (3.5)  in  Equation  (3.2)  gives 

f=  P(R)  =  -  x  p  +  1  x  q  =  2q, 

P 

and  this  probability  is  less  than  1.  When  q  >  p,  we  have 

/  =  P{R)  =1  x  p  +  -  x  q  =  2p, 

<7 

and  this  is  also  less  than  1.  And  when  p  =  q  =  we  have 
/  =  P(R)  =1  x  p  +  1  x  q  =  1. 

That  is, 

2  q  q  <  p 

/= . 1  q=p=\ 

2p  q  >  p 
l -(p-q)  q  <  p 

=  <  1  q  =  v  =  \ 

l  -(q-p)  q>  p, 
since  p  +  q  =  1 ;  more  simply, 

/=  1  -  \p  -  q\. 

Thus,  return  to  the  origin  is  certain  only  when  the  random  walk  is  symmetric — 
that  is,  when  p  =  q  =  and,  in  this  case,  the  random  walk  is  said  to  be  recurrent 
Notice  that  this  use  of  the  term  ‘recurrent’  is  consistent  with  its  literal  meaning — 
occurring  again  or  periodically — as  the  event  ‘the  particle  visits  the  origin’  is 
certain  to  occur  again. 

When  either  p  >  q  or  p  <  q,  the  particle  is  not  certain  to  return  to  the  origin;  there 
is  a  positive  probability  that  the  particle  will  never  return  and,  in  this  case,  the 
random  walk  is  said  to  be  transient. 

The  terms  recurrent  and  transient  are  not  reserved  exclusively  for  random  walks; 
they  are  used  to  classify  the  states  of  Markov  chains  in  general.  A  state  to  which 
eventual  return  is  certain  is  recurrent;  and  one  to  which  return  is  not  certain  is 
transient.  In  defining  the  terms  recurrent  and  transient  as  they  apply  to  random 
walks,  we  have  considered  only  returns  to  the  origin.  However,  for  a  random  walk, 
the  probability  of  eventual  return  to  state  1,  or  indeed  of  eventual  return  to  any 
other  state,  has  the  same  value  as  the  probability  of  eventual  return  to  the  origin, 
so  either  all  states  are  recurrent  or  all  are  transient;  so  it  is  usual  to  speak  of  a 
random  walk  as  a  whole  as  being  either  recurrent  or  transient.  The  recurrence  and 
transience  of  states  will  be  discussed  in  more  detail  in  Unit  6. 


This  form  for  /  will  be  derived  by 
another  method  in  Subsection  4.2. 
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3.2  The  position  of  the  particle 

You  have  just  seen  that  a  particle  executing  a  symmetric  random  walk  is  certain  to 
return  to  its  starting  point  sooner  or  later.  Does  this  mean  that  it  will  stay  near  its 
starting  point?  How  far  away  is  it  likely  to  wander? 

And  if  a  random  walk  is  not  symmetric,  there  is  a  positive  probability  that  the 
particle  will  never  return  to  the  origin.  So  where  is  the  particle  likely  to  be?  Does 
it  drift  further  and  further  away? 

We  shall  begin  to  answer  these  questions  by  finding  the  distribution  of  Xn,  the 
position  of  the  particle  after  n  steps;  this  is  closely  related  to  the  binomial 
distribution.  In  the  next  question  you  are  asked  to  find  the  distribution  of  Xn  for 
some  small  values  of  n;  try  this  question  before  you  read  the  derivation  for  general 
n  which  follows. 


Question  3.1  Suppose  that  a  particle  executing  a  simple  random  walk  (with 
p  +  q  =  1)  starts  at  the  origin. 

(i)  List  the  possible  positions  of  the  particle  after  (a)  1  step,  (b)  2  steps,  (c)  3  steps, 
(d)  4  steps. 

(ii)  Find  the  distribution  of  Xn,  the  position  of  the  particle  after  n  steps,  when 

(a)  n  =  1,  (b)  n  =  2,  (c)  n  =  3.  □ 

Two  things  are  evident  from  the  solution  to  this  question.  First,  from  part  (i),  it 
seems  that  after  an  even  number  of  steps  the  possible  positions  are  even  integers 
and  after  an  odd  number  of  steps  the  possible  positions  are  odd  integers;  this  is 
true  for  all  n  (as  you  will  see  shortly).  Second,  from  part  (ii),  it  is  clear  that,  as  n 
increases,  the  work  required  to  find  the  distribution  of  Xn  (for  any  particular  n) 
increases  rapidly;  so  a  general  result  will  be  useful. 

Suppose  that,  in  the  first  n  steps  of  a  simple  random  walk  starting  at  the  origin,  x 
steps  are  to  the  right  (+1)  and  y  steps  are  to  the  left  (- 1);  so  x  +  y  =  n  and 

X„  =  x  —  y  =  x  —  (n  —  x)  =  2x  —  n.  (3.6) 

For  convenience,  we  write  k  =  2x  —  n.  From  Equation  (3.6)  it  can  be  seen  that  if  n 
is  even  then  Xn  is  also  even,  and  if  n  is  odd  then  Xn  is  also  odd.  That  is,  after  an 
even  number  of  steps  the  possible  positions  correspond  to  even  integers  whereas 
after  an  odd  number  of  steps  the  possible  positions  correspond  to  odd  integers.  It 
follows  that  P{Xn  =  k)  is  zero  if  n  and  k  are  not  either  both  even  or  both  odd. 

Question  3.2  List  the  possible  positions  of  the  particle  after  n  steps,  (a)  if  n  is  odd, 
(b)  if  n  is  even.  □ 


Now  let  Wn  be  the  number  of  positive  steps  in  the  first  n  steps;  then  W„  has  a 
binomial  distribution,  W„  ~  B(n,  p ),  and,  for  0  <  x  <  n. 


P(Wn  =  x)  = 


But,  from  Equation  (3.6),  W„  =  x  if  and  only  if  Xn  =  2x  —  n,  so 


P(Xn  =  2  x-n)  =  rW 


Then,  since  2 x  —  n  =  k  (so  that  x  =  (n  +  k)/2),  we  have 


P(X„  =  k)  = 


(n  +  k)/2 


p(n  +  k)l2q(n-k)/2. 


(3.7) 


and  this  holds  for  k  =  ±  1,  ± 3, . . . ,  ±n  when  n  is  odd,  and  for 
k  =  0,  ± 2,  ± 4, . . . ,  ±n  when  n  is  even. 


Question  3.3  Use  Formula  (3.7)  to  find  the  following  probabilities  when  p  =  0.6. 
(i)  P(Xl0  =  4),  (ii)  P{Xl5  =  3),  (iii)  P(X20  =  -2).  □ 


A  table  of  values  of  binomial 
coefficients  is  contained  in  Neave, 
page  44. 
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Although  Formula  (3.7)  gives  the  probability  that  the  particle  is  in  any  particular 
position  after  n  steps,  it  is  not  easy  to  see  from  this  the  range  of  positions  within 
which  the  particle  is  most  likely  to  be.  However,  since  Xn  is  the  sum  of  the  n 
independent  and  identically  distributed  random  variables  Z1,Z2,  Z„,  we  can 

apply  the  Central  Limit  Theorem.  This  says  that,  for  large  n,  the  distribution  of  X„ 
is  approximately  normal: 


X„  E(Xn)^N{Qly 


(3.8) 


Thus,  once  we  have  found  the  mean  and  variance  of  the  particle’s  position,  we  can 
use  tables  to  calculate  the  approximate  probability  of  the  particle  being  in  any 
given  range  of  positions. 


Question  3.4 

(i)  Determine  the  values  of  £(Z,)  and  F(Z,)  in  terms  of  p  and  q. 

(ii)  Hence,  show  that 

E{Xn)  =  n(p  -  q)  and  V(Xn)  =  4 npq.  □ 

Question  3.5 

(i)  Find  the  maximum  value  of  the  function  g  given  by 

g(p)  =  4p(l  -  p),  0<p<l. 

(ii)  Write  down  in  terms  of  p  the  variance  of  the  position  after  n  steps  of  a  particle 
executing  a  simple  random  walk  with  p  +  q  =  1.  For  what  value  of  p  is  this 
variance  a  maximum?  □ 

The  solutions  to  these  two  questions  tell  us  quite  a  lot  about  the  general  behaviour 
of  a  random  walk.  Since  the  mean  position  of  the  particle  is  n(p  —  q),  when  p  >  q 
(that  is,  p  >  $)  the  particle  tends  to  drift  to  the  right  as  n  (time)  increases,  the  drift 
being  greater  the  larger  the  value  of  p — see  Figure  3.1(i). 


Figure  3.1  Mean  and  variance  for  p  >  \ 


However,  for  each  «  asp(>^)  increases,  the  variance  4 npq  diminishes — see 
Figure  3.1(ii).  So  a  particle  for  which  p  =  px  >  p2  >  \  drifts  more  to  the  right  than 
does  one  for  which  p  =  p2,  and  there  is  less  variability  in  its  position.  These 
remarks  are  illustrated  by  the  next  question. 


Question  3.6  Find  the  values  of  E(X„)  and  V(Xn)  when 
(i)  p  =  0.6,  (ii)  p  =  0.8,  (iii)  p  =  0.9.  □ 


By  substituting  in  Result  (3.8)  the  values  of  E(Xn)  and  V(Xn)  found  in  Question  3.4, 
we  obtain  the  following  result  for  a  simple  random  walk  with  p  +  q  =  1.  For 
large  n, 


Xn  ~  n(p  -  q) 

n/4 npq 


N{  0, 1). 


(3.9) 


This  enables  us  to  make  probability  statements  about  the  likely  position  of  the 
particle  after  a  long  time  or  to  find  the  probability  that  the  particle  is  within  any 
given  range  of  positions. 


When  p  <  q,  the  particle  tends  to 
drift  to  the  left. 
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Example  3.1 

Find  the  probability  that  the  particle  is  less  than  10  units  from  the  origin  after  25 
steps  if  p  =  0.8. 


Solution 


Using  the  result  of  Question  3.6(ii),  we  have  E(Xn)  =  0.6 n  and  V(Xn)  =  0.64 n,  so 
E{X25)  =  15  and  V(X2S)  =  16.  So,  by  Result  (3.8), 


N(0, 1). 


Using  this  normal  approximation  and  a  continuity  correction,  we  have 


p(- 10  <  x25  <  10)  ~  p(  95  -  15  <  Z  <  — - — 

\  4  4 

=  P(  — 6.125  <  Z  <  -1.375) 


=  <D(- 1-375)  —  <D(  — 6.125) 

~  0.085. 

So  the  particle  has  probability  0.085  (approximately)  of  being  less  than  10  units 
from  the  origin  after  25  steps.  □ 


Question  3.7  Find  the  approximate  probabilities  that  the  distance  of  the  particle 
from  the  origin  is  less  than  20  units  after  100  steps  if 

(i)  p  =  0.5,  (ii)  p  =  0.6,  (iii)  p  =  0.9. 

(In  each  case,  use  a  continuity  correction.)  □ 

The  Central  Limit  Theorem  can  be  used  in  the  following  way  to  make  probability 
statements  about  the  likely  position  of  the  particle.  If  Z  ~  N( 0, 1)  then,  for 
example, 

P(  —  1.96  <  Z  <  1.96)  =  0.95, 
so,  using  Result  (3.9), 

P(— 1.96  <  [Xn  -  n(p  -  qj]/yj4npq  <  1.96)  ~  0.95; 

that  is, 

P(n(p  -q)-  1.96^4 ^q  <  Xn  <  n{p  -  q)  +  1.96^4^)  ~  0.95.  (3.10) 

Hence,  the  probability  that  Xn  lies  between  n(p  —  q)  —  1 .96^/4 npq  and 
n(p  ~  q)  +  1 .96^/4 npq  is  approximately  0.95.  Similarly, 

P(-2.58<Z  <  2.58)  =  0.99, 
so 

P(n(p  -q)-  2.5Sy/4npq  <  Xn  <  n(p  -  q)  +  2.58^4^)  ~  0.99,  (3.1 1) 

and  the  probability  that  Xn  lies  between  n(p  -  q)  -  2.58^4 npq  and 
n(p  —  q)  +  2.58v/4 npq  is  approximately  0.99. 

Example  3.2 

For  a  simple  random  walk  with  p  =  0.8,  using  the  result  of  Question  3.6(ii),  we 
have  E(Xn)  =  0.6 n  and  V{Xn)  =  0.64n;  so  E(Xl00)  =  60  and  V(Xl00)  =  64. 
Substituting  these  values  in  Equations  (3.10)  and  (3.11),  we  obtain 

P(60  -  1.96n//64  <  X100  <  60  +  1.96^/64)  ~  0.95, 

P(60  -  2.58^/64  <  <  60  +  2.58^64)  ~  0.99. 

That  is,  P(44.32  <  X100  <  75.68)  ~  0.95  and  P(39.36  <  X100  <  80.64)  ~  0.99.  So, 
when  n  =  100  and  p  =  0.8,  the  particle  will  be  in  a  position  between  44  and  76 
(inclusive)  with  probability  0.95  (approximately);  and  it  will  be  between  40  and  80 
(inclusive)  with  probability  0.99  (approximately).  □ 

Question  3.8  When  n  =  100,  find  a  range  of  positions  within  which  the  particle 
will  be  with  probability  0.95,  approximately,  if  (i)  p  =  0.5,  (ii)  p  =  0.9.  □ 


Z  is  a  standard  normal  variate, 
Z  ~  N( 0, 1). 


The  only  possible  positions  for  the 
particle  after  100  steps  are  even 
integer  points,  so  the  ranges  given 
imply  only  even  integer  values.  For 
convenience,  each  extreme  point  of  a 
range  has  been  rounded  to  the 
nearest  even  integer. 
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Question  3.9  For  p  =  0.5,  find  an  approximate  interval  of  values  within  which  X„ 
will  be  with  probability  approximately  0.95  when  (i)  n  =  900,  (ii)  n  =  3600.  □ 


The  following  tables  summarize  Solutions  3.8  and  3.9. 

n  =  100  p  =  0.5 


Width  of  interval 

V 

Width  of  interval 

n 

yfn 

40 

0.5 

116 

900 

30 

24 

0.9 

236 

3600 

60 

These  tables  illustrate  the  fact  that  the  width  of  the  interval  of  likely  positions  is 
proportional  to  the  standard  deviation  jAnpq  =  JA n(p  -  p2).  (See  also  Equations 
(3.10)  and  (3.11).)  In  particular,  this  means  that 

(i)  the  width  of  the  interval  of  likely  positions  is  smaller  for  larger  p,  that  is  when 
there  is  a  greater  tendency  to  drift  to  the  right; 

(ii)  the  width  of  the  interval  of  likely  positions  is  proportional  to  s/nt  not  to  n. 


These  points  are  further  illustrated  in  Figure  3.2,  which  shows  the  mean  position 
and  the  positions  two  standard  deviations  from  the  mean  against  p,  for  n  up  to  25 
and  for  three  values  of  p. 
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(ii)  p  =  0.7 
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(iii)  p  =  0.9 


Figure  3.2  Mean  position  and  intervals  of  likely  positions 


The  ideas  of  this  subsection  are  illustrated  in  Band  C  of  the  video-cassette;  this 
contains  a  number  of  simulations  of  simple  random  walks  for  different  values  of  p. 
You  may  find  it  helpful  to  watch  this  now. 

We  conclude  this  section  with  a  question  which  uses  many  of  the  ideas  introduced 
in  this  subsection;  you  are  asked  to  apply  these  ideas  to  a  random  walk  in  which 
steps  of  sizes  2  and  —  1  are  possible. 

Question  3.10  In  an  unrestricted  random  walk  starting  at  the  origin,  the  ith  step, 
Zh  has  the  following  distribution: 

P(Zi  =  2)  =  p,  P(Zt  =  —l)  =  q=  l—p. 

(i)  Find  the  mean  and  variance  of  Z,. 

(ii)  Hence  find  the  mean  and  variance  of  Xn,  the  position  of  the  particle  after  n 
steps. 

(iii)  Derive  the  probability  distribution  of  Xn  by  using  a  method  similar  to  that 
used  in  establishing  Formula  (3.7),  which  specifies  the  distribution  of  Xn  for  a 
simple  random  walk. 

(iv)  When  p  =  i,  find  the  values  of  (a)  E(X20),  (b)  V(X20),  (c)  P(X20  =  0), 

(d)  P(X20  =  1). 

(v)  When  p  =  £,  find  the  approximate  value  of  P(  —  70  <  Xx  80  <  70).  □ 


Similar  results  hold  when  there  is  a 
drift  to  the  left,  that  is  when  p  <  q. 


25  n 
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4  Return  probabilities  and  generating  functions 


In  Subsection  3.1,  we  calculated  the  probability  of  eventual  return  to  the  origin  of 
a  particle  (which  started  at  the  origin)  executing  a  simple  random  walk  with 
p  +  q  =  1;  this  probability  is  1  if  the  random  walk  is  symmetric,  but  otherwise  it  is 
less  than  1.  Now  we  shall  consider  the  time  at  which  a  return  occurs.  There  are 
two  probabilities  of  interest;  these  are  defined  for  any  random  walk.  The  first  is 
the  probability  that  the  particle  returns  to  the  origin  for  the  first  time  at  time 
"  (-  1)~thls  is  denoted  by  /„;  the  second  is  the  probability  that  the  particle  visits 
the  origin,  not  necessarily  for  the  first  time,  at  time  n  (>0)— this  is  denoted  by  u„. 


un  =  P(Xn  =  0),  n  >  0, 

fn  ~  P{Xn  =  0  and  Xr  #  0  for  0  <  r  <  n),  n  >  2 


/i  =  P(X  i  =  0)  =  ttl. 

Since  X0  =  0,  we  have  u0  =  1,  by  definition;  but  /„  is  defined  above  only  for  n  >  1; 
for  convenience,  f0  is  defined  by  f0  =  0.  Notice  that  we  have  already  found  the 
value  of  u„,  where  n  is  even,  for  a  simple  random  walk  with  p  +  q  =  1-  this  is  given 
by  Formula  (3.7)  with  k  =  0.  Thus, 


un  =  P(Xn  =  0)  = 


n/2 


pnllqnl2  jf  n  js  eyen 


if  n  is  odd. 


(4.1) 


But  we  know  very  little  as  yet  about  /„.  If  R„  is  the  event  that  the  first  return  to 
the  origin  occurs  at  time  n,  then  R,  the  event  that  a  return  eventually  occurs  is 
given  by 


R  =  HjU  R2kj  ...u  R„u ...  . 

Since  the  events  RX,R2,  are  mutually  exclusive,  P(Rn)  =/„  and  P(R)  =  f  we 
have 


=  1 


f  —  fl  +  f 2  +  •  •  •  +  fn  +  • 

This  suggests  another  way  of  deciding  whether  a  random  walk  is  recurrent  or 
transient:  instead  of  finding  /  directly,  we  could  calculate  the  individual  first  return 
probabilities.  In  fact,  for  most  random  walks  it  is  quite  difficult  to  find  an  explicit 
expression  for  /„;  however,  it  is  straightforward  to  find  a  relationship  between  the 
return  probabilities,  u„,  and  the  first  return  probabilities, /„.  And,  by  introducing 
generating  functions,  we  can  derive  an  alternative  criterion  for  deciding  whether  a 
random  walk  is  recurrent  or  transient;  this  involves  the  return  probabilities,  un. 
This  may  seem  surplus  to  our  requirements,  but  for  more  complicated  random 
walks  it  is  much  easier  to  use  this  alternative  criterion  than  it  is  to  calculate  / 
directly.  And,  as  you  will  see  in  Unit  6,  the  method  can  be  applied  to  Markov 
chains  m  general.  We  shall  conclude  this  section  by  using  the  main  result  on 
generating  functions  to  find  /„  for  the  special  case  of  a  simple  random  walk  with 
P  +  q=  1. 
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4.1  Generating  functions 


We  begin  by  considering  two  series: 

00 

F(s)  =  £  f,sn  =/o  +/i5  +f2s2  +  ...  +  /„s"  +  ... 

n  =  0 

and 

CO 

t/(5)  =  X  W„s"  =  U0  +  UXS  4-  «2s2  +  •••  +  + - 

n  =  0 

Since  the  coefficients  in  these  series  are  probabilities,  they  are  between  0  and  1  and 
thus  the  series  both  converge  for  |s|  <  1  and  so  define  functions:  F(s)  and  U(s)  are 
called  generating  functions.  But  notice  that  although  U(s)  generates  a  set  of 

probabilities,  it  may  not  be, the  generating  function  of  a  probability  distribution: 

00 

the  sum  of  the  probabilities  £  un  is  not,  in  general,  equal  to  1;  so  {«„:  n  >  0}  is 

71  =  0 

not,  in  general,  a  probability  distribution.  For  any  random  walk,  u0  =  1  and,  for 
example,  for  a  simple  random  walk  un>  0  for  all  even  integers  n,  so  the  sum  of  the 
probabilities,  un,  is  greater  than  1.  Because  of  this,  we  have  referred  to  U(s)  as  a 
generating  function  and  not  as  a  probability  generating  function,  so  that  you  will 
not  confuse  U(s)  with  the  p.g.f.’s  of  Unit  4,  which  have  properties  which  U(s)  does 
not ,  in  general,  possess. 

But  is  F(s)  a  p.g.f.?  To  answer  this,  we  must  look  at  the  sum  of  probabilities 
/i  +  fi  +  ,  which  is  equal  to  /.  If  the  random  walk  is  recurrent,  then 

/=  1,  and  so  F(s)  is  the  p.g.f.  of  the  time  at  which  the  first  return  occurs;  but  if  the 
random  walk  is  transient,  then  /<  1,  and  in  this  case  F(s)  is  not  the  generating 
function  of  a  proper  probability  distribution — there  is  a  positive  probability  that 
the  time  at  which  the  first  return  occurs  is  infinite. 

Our  main  objective  in  this  section  is  to  find  a  simple  relationship  between  U{s)  and 
F{s)  and  to  deduce  an  alternative  criterion  for  recurrence  (in  terms  of  un).  To  do 
this  we  shall  first  build  up  a  picture  of  the  way  the  individual  probabilities  are 
related  to  one  another;  we  shall  do  this  for  any  random  walk  (not  just  for  a  simple 
random  walk  with  p  +  q  =  1). 

We  have  u0  =  1,  and  we  know  that  ul=fXi  but  what  about  u2?  This  is  the 
probability  that,  starting  from  the  origin,  the  particle  is  again  at  the  origin  after 
two  steps.  There  are  two  ways  in  which  this  can  happen:  either  the  particle  returns 
for  the  first  time  after  one  step  and  then  returns  again  after  one  more  step,  or  the 
particle  returns  for  the  first  time  after  two  steps.  These  two  events  are  mutually 
exclusive,  so  u2  is  equal  to  the  sum  of  their  probabilities.  The  first  event  has 
probability  fx  x  ux  and  the  second  has  probability  f2,  so 

U2  =/lMl  +fi- 

Since  u0  =  1  and  f0  =  0,  we  can  rewrite  this  as 

u2  =fou2  +flUl  +/2M0> 
which  is  symmetric  in  us  and  /s. 

Question  4.1 

(i)  Show  that 

“3  =/o“3  +/lW2  +/2«1  +/3M  0, 
and  find  a  similar  expression  for  u4. 

(ii)  Write  down  an  expression  for  u„.  □ 

By  considering  the  time  at  which  the  first  return  occurs,  you  should  have  obtained 
=/o“*  +/i««-i  Ff2un-2  +  •••  +/„-i«i  +/„w0.  (4.2) 

We  shall  use  this  identity  to  establish  the  main  result  of  this  section,  that 


U(s)  =  1  +  F(s)  U(s). 


(4.3) 


0  =  0  by  definition. 
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Proof 

We  shall  do  this  by  looking  at  the  terms  involving  s",  for  n  >  0,  on  each  side  of 
Identity  (4.3).  First  look  at  the  product  F(s)  U(s): 

F(s)U(s)  =  (/0  +flS  +f2s2  +  ...  +frsr  +  ...)(u0  +  u  iS  +  ...  +  umsm  +  ...). 
We  shall  pick  out  the  term  in  s"  for  n  >  1;  this  is  made  up  of  the  sum  of  products 
of  pairs  of  terms,  one  from  each  bracket,  obtained  as  follows:  the  constant  term 
from  the  first  bracket  multiplied  by  the  term  in  s"  from  the  second;  the  term  in  s 
from  the  first  bracket  multiplied  by  the  term  in  sn_1  from  the  second;  the  term  in 
s2  from  the  first  bracket  multiplied  by  the  term  in  s"~2  from  the  second;  and  so 
on.  Thus,  the  term  in  s"  is 

/0  X  UnSn  +f,S  X  +  f2s2  x  un-  2s"~ 2  +  ...  +fnsn  x  u0. 

This  is  equal  to 

(foK  +fiK-i  +fiun~2  +  •••  +f„u0)sn  =  unsn 
using  Identity  (4.2).  But,  for  any  n  >  1,  this  is  the  term  involving  s"  in  U{s),  the  left- 
hand  side  of  Identity  (4.3).  It  remains  only  to  look  at  the  terms  independent  of  s: 
on  the  right-hand  side  this  term  is  1  +  f0u0,  which  equals  1  since  /0  =  0;  and  on 
the  left-hand  side  it  is  u0,  which  is  also  equal  to  1.  Hence  the  two  sides  of  Identity 
(4.3)  are  equal,  thus  establishing  the  result.  ■ 


An  alternative  criterion  for  recurrence 


We  are  now  very  close  to  obtaining  the  desired  alternative  criterion  for  recurrence. 
Rearranging  Identity  (4.3)  gives 

{u(s) # 0); 

that  is, 


The  generating  functions  have  been  defined  only  for  \s\  <  1,  but  ignoring  any 
complications  and  letting  s  tend  to  1  in  this  last  expression,  we  obtain: 

Z/n  =  1  -  J-; 

and  so  £/„  =  1  if  and  only  if  £  un  is  infinite. 

So  we  have  the  following  criterion  for  the  recurrence  of  a  random  walk.  (A  result 
equivalent  to  this  for  Markov  chains  is  considered  in  Unit  6.) 


When  the  limits  are  omitted  from  a 
sum,  as  here,  it  is  assumed  that  the 
sum  is  for  values  of  n  as  declared 
previously. 


A  rigorous  proof  of  this  result 
requires  mathematics  beyond  the 
scope  of  this  course. 


A  criterion  for  the  recurrence  of  a  random  walk 

A  random  walk  is  recurrent  if  and  only  if  the  sum  of  the  return  probabilities 
is  infinite: 

/=  1  if  and  only  if  =  oo. 


4.2  First  return  probabilities 


In  this  subsection  generating  functions  will  be  used  to  confirm  the  result  of 
Subsection  3.1,  that  a  simple  random  walk  with  p  +  q  =  1  is  transient  when  p  ^  q 
and  recurrent  when  p  =  q  =  And,  by  an  application  of  the  generating  function 
identity  (4.3),  the  first  return  probabilities  will  be  found  in  this  case. 


The  return  probabilities  are  given  by  Formula  (4.1);  when  n  is  odd  u„  is  zero,  and 
when  n  is  even,  n  =  2m  say,  the  return  probability  is  given  by 


= 


pmqm,  m  =  0,1,2,.... 


m 


(4.4) 
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So  we  can  write  down  an  expression  for  the  generating  function  U(s): 

u(s)=  e  (2:Wm  (4.5) 

To  write  this  in  a  concise  form,  we  shall  need  to  make  use  of  the  Binomial 
Theorem;  in  its  general  form  this  states  that,  for  any  rational  number  n, 

(1  +  x)n  =  1  +  nx  +  l)x2  +  ... 

+  n(n-  1)  x  ...  x  (n-r+  l)*r  +  ,  (46) 

provided  that  —  1  <  x  <  1.  (This  result  will  be  used  first  when  n  is  —  £.) 

The  term  in  Equation  (4.5)  involving  s2m  is 

2m\  2  (2m)!  2m 

pqs2  =  ——  pas2, 
m )  m!m! 

Before  the  Binomial  Theorem  can  be  used,  this  must  be  rearranged  so  that  the 
numerator  is  a  product  of  terms  decreasing  by  1  each  time,  and  the  denominator  is 
simply  ml.  Rewriting  this  term,  we  have 

ulms2m  =  pmqms2m 


2m(2m  —  l)(2m  —  2)  x  ...  x  3  x  2  x  1  t  2 


m!m! 

m(m  —  i)(m  —  1)  x  . . .  x  §  x  1  x| 


( Pqs2T 


mlml 


(4  pqs 


2 \m 


(m  -  i )(m  -  f)  x  ...  x  f  x  i 


ml 


(4  pqs 


2  \m 


(-i)(-i  -  1)  x  ...  x  (-i  -  m  +  2)(-i  -  m  +  1),  „  2^m 

= - - (-4  pqs2?- 

Thus,  using  the  Binomial  Theorem  (4.6),  we  see  that  Equation  (4.5)  can  be  written 
as 


U(s)  =  (1  -4  pqs2)~1/2. 

Then  Identity  (4.3)  gives 

F{s)  =  1  -  (1  -  4 pqs2)112.  (4.7) 

But  /=  YJh  is  given  by  F(  1),  so 
/=  i-(i-4Pqyi2 
=  1  -  ((p  +  <?)2  -  4pt?)1/2 
=  i  -  \p-ql 

as  found  in  Subsection  3.1.  Hence,  if  p  =  q,  then  /  =  1  and  the  random  walk  is 
recurrent;  and  if  p  #  q,  then  /  <  1  and  the  random  walk  is  transient. 


To  find  the  first  return  probabilities  /„,  we  expand  the  expression  (4.7)  for  the 
generating  function  F(s)  as  a  power  series.  You  are  asked  to  do  this  in  the  next 
question;  you  will  need  to  use  the  Binomial  Theorem,  but  this  time  for  n  =  j.  The 
algebraic  manipulation  involved  in  this  question  is  quite  difficult,  so  do  not  worry 
if  you  have  trouble  doing  it;  you  will  not  be  expected  to  reproduce  the  solution, 
but  you  should  be  able  to  apply  the  result. 


Question  4.2  Show  that  the  term  in  s2m  in  the  expansion  of  Formula  (4.7)  for  F(s) 
is  equal  to 


1  2m 


2m  —  1  \ 


m 


pqs 


Deduce  that  the  first  return  probabilities  are  given  by 


Am  — 


2  m\  1 


l2m- 


□ 


(4.8) 


So,  in  this  special  case  of  a  simple  random  walk  with  p  +  q  =  1,  there  is  a  very 
simple  relationship  between  the  return  probabilities  un  and  the  first  return 
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probabilities  /„.  However,  in  many  cases,  there  is  not  such  a  simple  relationship 
between  the  two  probabilities.  This  section  concludes  with  two  straightforward 
questions  in  which  you  are  asked  to  find  some  first  return  probabilities. 

Question  4.3  Find  the  probability  that  the  first  return  to  the  origin  in  a  simple 
random  walk  with  p  =  q  =  \  occurs  (i)  after  4  steps,  (ii)  after  10  steps, 

(iii)  after  20  steps.  □ 

Question  4.4 

(i)  Find  the  probability  that  the  first  return  to  the  origin  in  a  simple  random  walk 
with  p  =  0.6,  q  =  0.4,  occurs  after  6  steps. 

(ii)  Find  the  probability  that  the  first  return  to  the  origin  in  a  simple  random  walk 
with  p  =  0.3,  q  =  0.7,  occurs  after  12  steps.  □ 


5  Random  walks  with  reflecting  barriers 

5.1  Limiting  distributions 

We  return  to  the  gambling  scenario  of  Section  2  in  which  Gary  plays  a  series  of 
games  with  probability  p  of  winning  each.  He  begins  with  £j  and  his  opponent 
with  £(a  —j),  and  they  risk  £1  on  the  outcome  of  each  game.  However,  now 
suppose  that  the  two  players  are  playing  solely  for  the  pleasure  of  the  game  and 
have  agreed  that  every  time  one  of  them  loses  his  last  pound  his  opponent  will 
immediately  return  it  so  that  the  game  can  continue. 

This  means  that  neither  player  can  ever  be  totally  ruined;  the  game  can  continue 
indefinitely.  So  there  is  a  different  type  of  long-term  behaviour  from  that  in  the 
absorbing-barrier  case,  and  the  questions  we  might  ask  about  this  situation  are 
different.  It  no  longer  makes  sense  to  talk  about  the  probability  of  ruin  and  the 
expected  duration  of  the  match.  Instead,  we  shall  ask  questions  such  as  ‘In  the 
long  run,  what  proportion  of  the  time  is  Gary’s  fortune  equal  to  £1, £2,  ..., 

£(a  -  1)?’  or  ‘What  is  Gary’s  average  fortune  over  a  long  period?’  or  ‘If  Gary  is 
down  to  his  last  pound,  how  long  can  he  expect  to  wait  before  he  increases  his 
fortune  to  £2?’. 

Before  looking  at  these  questions  we  shall  identify  the  gambling  situation  described 
above  with  a  simple  random  walk  with  two  reflecting  barriers.  As  before,  Xn 
represents  Gary’s  fortune  after  n  bets;  Z„  is  his  gain  on  the  nth  bet. 

The  only  difference  between  this  process  {*„;  n  =  0, 1,2,  ...}  and  the  one  in  the 
gambler’s  ruin  is  in  the  behaviour  in  the  barrier  states.  For  the  gambler’s  ruin  the 
barrier  states  are  0  and  a  and  these  are  absorbing  states — once  entered  they 
cannot  be  left.  In  the  present  situation  the  states  0  and  a  are  never  entered:  the 
players  have  ensured  by  their  agreement  that  they  will  each  always  have  at  least 
£1.  So  the  possible  states  of  Gary’s  fortune  are  1,2, ...,  a  -  1;  the  boundary  states 
are  1  and  a  —  1. 

As  long  as  Gary’s  fortune  is  not  either  £1  or  £{a  -  1)  then,  on  the  next  bet,  he  will 
win  £1  with  probability  p  or  lose  £1  with  probability  q.  But  if  at  any  time  his 
fortune  goes  down  to  £1  or  up  to  £(a  -  1)  then  the  rules  change.  For  example,  if 
he  has  only  £1  after  n  -  1  bets— that  is,  if  Xn_y  =  1— then  he  has  probability  p  of 
winning  the  next  bet  and  increasing  his  fortune  to  £2  (in  which  case,  Xn  =  2)  and 
probability  q  of  losing  the  next  bet.  But  if  he  loses  the  bet  his  opponent 
immediately  returns  his  money  so  his  fortune  is  still  £1— that  is,  Xn  =  1.  So  when 
Ar„_1  =  1,  the  distribution  of  Z„  is  given  by 

P(Z„=l\Xn^  =  l)  =  p, 

P(Zn  =  0\Xn_1  =  l)  =  q=l-p. 

This  is  illustrated  in  Figure  5.1. 

Question  5.1  Find  the  distribution  of  Z„  when  Xn_^  =  a  -  1,  and  draw  a  diagram 
to  illustrate  it.  □ 
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Figure  5.1  The  lower  reflecting 
barrier 


This  process  {X„;  n  =  0, 1,2,  ...}  is  a  simple  random  walk  with  two  reflecting 
barriers.  In  this  example  the  ‘reflections’  are  immediate— the  particle  never 
actually  moves  to  0  or  a;  if  a  step  would  take  the  particle  to  one  of  these  positions, 
then  the  particle  is  ‘held’  in  its  current  position  (1  or  a  —  1). 

We  are  now  nearly  ready  to  answer  the  questions  posed  earlier  in  this  section;  but 
first  we  need  some  additional  notation.  For  1  <  j  <  a  —  1,  let 

Pil  =  P{ Gary’s  fortune  is  £k  at  time  n  having  started  with  £j) 

=  P(Xn  =  k\X0=j). 

We  shall  use  the  Theorem  of  Total  Probability  to  obtain  an  equation  connecting  Page  11 

the  probabilities  pJJ,  pfc-l  and  This  time,  instead  of  conditioning  on  the 

outcome  of  the  first  bet  (as  in  the  gambler’s  ruin),  we  shall  condition  on  the 

outcome  of  the  nth  bet;  that  is,  the  bet  at  time  n.  For  1  <  k  <  a  —  1,  the  outcome 

of  the  nth  bet  is  either  a  win  with  probability  p— resulting  in  an  increase  of  £1  in 

Gary’s  fortune  from  £{k  -  1)  at  time  n  -  1  to  £k  at  time  n— or  a  loss  with 

probability  <?— reducing  his  fortune  from  £(/c  +  1)  at  time  n  -  1  to  £k  at  time  n. 

Thus, 

P(&  =  P{Xn  =  k\Xn_1=k-  1  )P(Xn.1  =  k  -  l\X0  =j) 

+  P(Xn  =  k\Xn-1  =  k+l)P(Xn.1=k+l\X0=j) 

=  pP{X„.i  =k  -  l\X0  =  j)  +  q  P(Xn  _  j  =k  +  1 1 AT0  =  j). 

That  is,  for  1  <  k  <  a  —  1, 

Pjll  =  P  Pjlk-l  +  4  Pf.k  +  l  •  (5.1) 

You  are  asked  to  consider  k  =  1  and  k  =  a  —  1  in  the  next  question. 

Question  5.2  Show  that 

PiR=qpij:r1)  +  qp^l) 

and  (5.2) 

p%-i  =  pp$;-'l  +  pph--l  □ 

The  first  question  asked  at  the  beginning  of  this  section  was  ‘In  the  long  run,  what 
proportion  of  time  is  Gary’s  fortune  equal  to  £1,£2, ...,  £{a  -  1)?’.  This  is 
equivalent  to  asking  what  proportion  of  the  time  each  of  the  states  1,2,  ...,  a  -  1 
is  occupied  by  a  particle  executing  a  random  walk  as  described  above.  This  can  be 
answered  by  finding,  for  each  k,  the  state  occupation  probabilities  pfl  for  large  n. 

Now  it  can  be  shown  that  for  a  random  walk  such  as  this  one  these  probabilities 
‘settle  down’  as  n  -»■  oo;  that  is, 

lim  pH  =  nk  for  1  <  k  <  a  —  1. 

n  -*  co  ’ 

Furthermore,  the  values  nk  are  independent  of  the  initial  state  j;  that  is,  however 
little  or  much  money  Gary  begins  with,  after  a  large  number  of  bets,  his 
probability  of  having  £k  is  nk.  These  probabilities  nk,  1  <  k  <  a  -  1,  give  the 

limiting  distribution  of  the  state  occupied;  nk  is  the  proportion  of  time  spent  in  Limiting  distributions  are  discussed 

state  k  in  the  long  run,  or  the  proportion  of  the  time  that  Gary’s  fortune  is  £k.  in  more  detail  in  Unit  6. 

It  remains  only  to  find  the  values  of  the  probabilities,  nk;  to  do  this,  look  again  at 
Equations  (5.1)  and  (5.2).  Letting  n  -►  oo  in  Equation  (5.1)  gives,  for  1  <  k  <  a  -  1, 

nk  =  piik-x  +  qnk  +  l.  (5.3) 

And  Equations  (5.2)  lead  to 
=  qn  1  +  qn2, 

na-  1  —  Pna-2  +  P^fl-l 

Equation  (5.4)  reduces  to 

*2  =  ^,;  (5.6) 

and  Equation  (5.3)  with  k  =  2  is 

n2  =  pn1  +  qn3.  (5.7) 


(5.4) 

(5.5) 
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Question  5.3 

(i)  Show  that  n3  =-n2. 

q 

(ii)  Find  a  similar  formula  for  and,  by  extending  the  result  to  nk,  show  that  nk 
may  be  written  as 

%  =  Ti\  for  1  <  k  <  a  —  1.  □ 


We  require  the  solution  of  the  equations  to  be  a  probability  distribution;  so  we 
require 

ni  +  n2  +  •••  +  na-i  =  i; 

that  is, 


i+5+ 


+ 1  - 


n-- 


using  Question  5.3(ii). 


The  term  in  the  square  brackets  is  the  sum  of  a  -  1  terms  of  a  geometric  series  (or 
G.P.)  with  common  ratio  p/q;  so,  when  p  ^  q. 


n 


i 


1  -p/q  ) 


that  is, 


1; 


Til 


giving 


1  -  p/q 

1  -ip/qr1’ 


nk 


j-p/q  (p\  1 

i  -w1  W 


for  1  <  k  <  a  -  1. 


When  p  >  q,  as  k  increases  from  1  to  a  —  1,  nk  increases  geometrically;  whereas, 
when  p  <  q,nk  decreases  geometrically  as  k  increases  from  1  to  a  -  1.  So,  the 
proportion  of  time  that  Gary  has  £/c  increases  or  decreases  geometrically  with  k 
according  as  p  is  greater  than  or  less  than  q.  The  limiting  distribution  is  illustrated 
in  Figure  5.2  when  a  =  10,  for  p  =  0.6  and  p  =  0.4. 


Figure  5.2  (i)  The  limiting  distribution  when  a  =  10,  p  =  0.6; 

(ii)  the  limiting  distribution  when  a  =  10,  p  =  0.4 

Question  5.4  Show  that,  when  p  =  q  = 

%  =  — ~r  for  1  <  k  <  a  —  1.  □ 

a  —  1 
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Now  that  we  have  found  the  limiting  distribution  we  can  answer  the  second  of  our 
questions:  Gary’s  average  fortune  is  the  mean  of  this  distribution.  You  are  asked 
to  find  this  in  the  next  question. 


Question  5.5 

(i)  Find  Gary’s  average  fortune  when  p  =  q. 

(ii)  Show  that  Gary’s  average  fortune  is 

1  (a  —  l)xa_  1 


1  -x 
where  x  =  p/q. 


1  -xfl_1 
□ 


P  *  <b 


(5.8) 


To  see  what  the  result  in  Question  5.5  means  in  practice,  we  shall  look  at  the 
value  of  the  mean  when  a  is  large.  If  p  >  q,  then;  using  Result  (5.8),  it  follows  that 
Gary’s  average  fortune  is  approximately  a  -  1  -  1  /(p/q  -  1).  If  p  =  f ,  for  example, 
then  his  average  fortune  is  £(a  —  2);  so  he  will  have  most  of  the  money  for  much 
of  the  time.  Similarly,  it  can  be  shown  that  if  a  is  large  and  p  <  q,  then  his  average 
fortune  is  1/(1  —  p/q)  and  so  he  will  have  very  little  money  most  of  the  time — his 
average  fortune  in  the  case  p  =  ^  would  be  £2. 

Our  final  question  was  ‘If  Gary  is  down  to  his  last  pound,  how  long  can  he  expect 
to  wait  before  he  increases  his  fortune  to  £2?’.  That  is,  what  is  the  average  length 
of  time  spent  in  state  1  before  moving  to  state  2? 

Let  Tbe  the  length  of  Gary’s  visit  to  state  1 — this  is  the  number  of  bets  at  state  1; 
the  possible  values  of  T  are  1,2,  ...  (because  he  stays  at  state  1  while  he  loses).  The 
probability  that  T  is  1  is  the  probability  that  he  wins  the  first  bet  after  his  fortune 
goes  down  to  £1,  and  this  is  p.  The  probability  that  Tis  t  (t  >  1)  is  the  probability 
that  he  loses  t  —  1  bets  and  then  wins,  that  is, 

P(T=t)  =  q'~1p,  t  =  1,2, ...  . 

So  Thas  a  geometric  distribution,  T~  Gt(p);  this  has  mean  E(T)  =  1/p,  which  is 
the  average  length  of  time  spent  in  state  1. 


Question  5.6  Suppose  that  Gary  has  £(a  -  1).  How  long  can  he  expect  it  to  be 
before  his  fortune  decreases  to  £(a  —  2)?  □ 


In  Band  D  of  the  video-cassette,  this  gambling  situation  is  simulated  and  the  idea 
of  a  limiting  distribution  is  illustrated.  Now  would  be  a  suitable  time  to  watch  this. 


5.2  Other  types  of  random  walks 

This  section  concludes  by  describing  briefly  a  model  for  the  content  of  a  reservoir. 
This  provides  another  example  of  a  random  walk  with  reflecting  barriers  but,  as 
you  will  see,  in  one  respect  it  is  quite  different  from  the  other  random  walks  that 
you  have  met  in  this  unit. 

Example  5.1  Content  of  a  reservoir 

The  level  of  the  water  in  a  reservoir  is  recorded  at  regular  intervals— for  example, 
at  the  end  of  each  day — and  X„  represents  the  amount  of  water  (measured  in 
suitable  units)  in  the  reservoir  at  time  n  (n  =  0, 1,2, ...).  The  amount  of  water 
flowing  into  the  reservoir  on  the  nth  day  is  Y„;  this  is  a  random  quantity — it 
depends  on  the  rainfall  and  thus  on  the  level  of  the  rivers  and  streams  feeding  the 
reservoir.  So  {Xn;  n  =  0, 1,  ...}  is  a  random  process. 

On  most  days  a  fixed  quantity  of  water  is  drawn  from  the  reservoir;  however, 
during  a  dry  spell  there  may  not  be  sufficient  water  in  the  reservoir  and,  in  this 
case,  it  becomes  drained.  Also  C,  the  capacity  of  the  reservoir,  imposes  a  natural 
upper  limit  on  the  amount  of  water  retained  in  the  reservoir  at  the  end  of  each 
day.  So  during  an  exceptionally  wet  spell  it  will  overflow;  that  is,  a  larger  quantity 
than  normal  will  be  released. 


In  part  (ii)  you  will  need  to  evaluate 

a-  1 

the  sum  S  =  £  kxk~\  This  may  be 

k=  1 

obtained  by  writing  down  the  series 
S  and  xS,  forming  (1  —  x)S  and  then 
dividing  through  by  (1  —  x). 


1  { I  >  1 »  1 
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If  we  assume  that  the  random  variables  Yn  (n>  1),  representing  the  inflows,  are 
independent  and  identically  distributed  random  variables,  then  the  process 
{X„;n  =  0,1,2,...}  is  a  random  walk  on  the  interval  [0,  C]  with  reflecting  barriers 
at  0  and  C. 

In  all  the  examples  that  you  have  met  in  the  earlier  sections  of  this  unit,  the  steps 
of  the  random  walk  were  unit  steps  or  integer  steps,  but  this  is  not  necessarily  so 
here.  The  quantity  Y„  of  water  flowing  into  the  reservoir  on  day  n  is  a  continuous 
random  variable,  so  the  change  in  the  content  of  the  reservoir  is  also  a  continuous 
random  variable;  and  the  states  of  the  random  walk  are  the  real  numbers  in  the 
interval  [0,  C]. 

Although  this  process  is  different  from  the  random  walk  with  reflecting  barriers  in 
the  gambling  problem,  many  of  the  questions  we  might  ask  about  it  are  the  same; 
for  example,  we  might  be  interested  in  the  long-run  or  limiting  distribution  of  Xn, 
the  content  of  the  reservoir;  or  in  the  duration  of  empty  periods  and  full  periods — 
periods  spent  on  one  of  the  barriers.  □ 

This  example  illustrates  one  way  in  which  random  walks  may  differ  from  the 
simple  random  walks  which  have  been  the  subject  of  most  of  this  unit.  There  are 
many  other  possibilities.  For  example,  the  simple  random  walk  on  the  line 
described  in  Section  3  can  be  generalized  by  allowing  steps  of  other  sizes: 

2, 3,  — ,  /c,  — ;  it  is  then  usually  referred  to  as  a  general  random  walk  on  the  line. 

(In  Question  3.10,  steps  of  size  2  were  allowed.)  Or  steps  in  other  directions  may 
be  allowed.  For  example,  the  particle  may  move  in  the  plane  taking  unit  steps  in 
the  plane  parallel  to  the  axes  as  shown  in  Figure  5.3;  this  is  called  a  simple  random 
walk  in  the  plane.  Again  steps  of  other  sizes  may  also  be  allowed,  or  diagonal 
steps;  or  the  particle  may  be  able  to  move  in  three-dimensional  space.  Indeed, 
much  work  has  been  done  on  random  walks  in  ^-dimensional  space,  where  d  may 
be  greater  than  three!  None  of  these  are  considered  in  this  course,  but  you  should 
be  aware  that  simple  random  walks  are  only  a  small  part  of  a  much  broader 
subject. 


Objectives 


After  studying  this  unit  you  should  be  able  to: 

explain  the  meanings  of  the  terms  random  walk,  simple  random  walk,  Markov 
chain,  transition  probability,  absorbing  and  reflecting  barriers,  recurrent  and 
transient  as  applied  to  an  unrestricted  random  walk; 

describe  the  classical  gambler’s  ruin  problem,  derive  and  solve  the  recurrence 
relations  for  the  probability  of  ruin  and  the  duration  of  the  match;  deduce  the 
corresponding  results  for  gambling  against  an  opponent  with  unlimited  resources; 
and  apply  the  results  of  the  gambler’s  ruin  problem  to  find  the  probability  of 
eventual  return  to  the  origin  of  a  particle  executing  a  simple  random  walk  on  the 
line; 

use  the  Central  Limit  Theorem  to  obtain  information  about  the  likely  range  of 
positions  at  time  n  of  a  particle  executing  a  random  walk  on  the  integers; 

use  generating  functions  to  obtain  a  relationship  between  the  return  probabilities 
and  first  return  probabilities  in  a  random  walk; 

explain  what  is  meant  by  the  limiting  distribution  of  a  random  walk  with  reflecting 
barriers; 

derive  and  solve  the  recurrence  relation  for  a  limiting  distribution  in  simple  cases. 


y , 

Pi 

A 

- 


<?2 


X 

Figure  5.3  The  possible  steps  for  a 
simple  random  walk  in  the  plane 
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Appendix:  Solutions  to  questions 


Section  1 


Section  2 


I.l  The  state  space  is  the  set  of  integers,  as  it  is  possible 
for  every  point  which  is  a  whole  number  of  units  from  the 
origin  (the  starting  point)  to  be  visited  eventually. 

1.2(i)  P  =  P(Zn  =  1)  =  P(two  heads)  =  i, 
q  =  P(Z„  =  —  1)  =  P(two  tails)  =  i 
and  r  =  P(Zn  =  0)  =  P(one  head)  =  £  +  i  =  i- 

(ii)  There  are  many  ways  of  assigning  the  random  digits  to 
outcomes;  one  possibility  is 
0,1, 2, 3, 4:  tail  (t), 

5, 6, 7, 8, 9:  head  (h). 

Then  the  first  two  digits  in  row  31  are  used  to  simulate 
tossing  the  two  coins  for  the  first  time;  these  are  59,  so  the 
outcome  is  two  heads,  which  means  Zx  is  1.  Using  the  next 
eighteen  digits  in  pairs  gives  the  following  realization. 


n 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Random  digits 

59 

99 

85 

00 

22 

98 

40 

95 

42 

61 

Outcome 

hh 

hh 

hh 

tt 

tt 

hh 

tt 

hh 

tt 

ht 

zn 

1 

1 

1 

-1 

-1 

1 

-1 

1 

-1 

0 

xn 

0 

1 

2 

3 

2 

1 

2 

1 

2 

1 

1 

1.3  The  state  space  is  {-50,-49,  —48,  ...};  that  is, 
(xeZ:  x  >  -50}. 


2.1(0  To  obtain  p}  we  must  replace  p  by  q  (and  q  by  p) 
and  j  by  a  —  j  in  the  expressions  for  q}  in  Frame  5A.  So 
(a)  when  p  =  q, 


(b)  when  p  ¥=  q, 

(p/qY~j  -  (p/q)a  _  ( q/p)j  ~  1 
j  1  -  (p/qY  ( q/pY  ~  1 

_  i  -  (q/p)J 

i  -  ( q/pY ' 

(ii)  When  p  =  q. 


pj  +  qj  =  - 
1  '  a 


a 


When  p  ^  q, 

_  .  _  1  -  ( q/p)j  ,  (q/p)J  -  (q/pY  t 

Pj  Qj  1  ~(q/pY  1  ~(q/pY 

Hence,  one  or  other  player  is  certain  to  be  ruined  eventually, 
so  the  match  is  certain  to  end. 


2.2(i)  The  situation  is  equivalent  to  the  gambler’s  ruin  with 
p  =  q  =  i,  j  =  10  and  a  =  30. 

(ii)  From  Frame  5 A,  the  probability  that  Gary  loses  the 
match  is  given  by 

n  -  1  _  12  _  2 
9l0  —  1  30  —  3- 


1.4(i)  The  second  absorbing  barrier  is  at  20. 

(ii)  The  state  space  is  (  —  50,  —49,  —48,  ...,  18,19,20};  that 
is,  {xeZ:  —50  <  x  <  20}. 


(iii)  Gary  loses  £10  with  probability  §  or  gains  £20  with 
probability  },  so  his  expected  loss  is 
10  x  $  -  20  x  J  =  0. 


(iii)  A  diagram  for  this  random  walk  is  given  below. 


1.5(i)  The  state  space  is  (0, 1,2,3}. 


(a) 

P(*n 

= 

l 

|*.-i 

=  0)  = 

=  1. 

(b) 

P(Xn 

= 

l 

\X-i 

=  1)  = 

P(K 

= 

2 

Xn  - 1 

=  1)  = 

-i 

(c) 

P(Xn 

= 

2 

*-l 

=  2)  = 

•i 

P(Xn 

= 

3 

Xn-! 

=  2)  = 

-i 

(d) 

P(  xn 

= 

3 

X„-t 

=  3)  = 

=  l. 

(No  other  transitions  are  possible.) 
(iii)  State  3  is  an  absorbing  state. 


2.3  This  is  the  gambler’s  ruin  problem  with  p  =  0.4;  so 
q/p  =  1.5. 

(i)  For  £1  bets,  a  =  100  and  j  =  10;  so  the  probability  of 
Gary’s  ruin  is 

(1.5)10  —  (1.5)100 
q'°~  1  —  (1.5)100 


(ii)  For  £5  bets,  a  —  20  and  j  =  2,  since  Gary  begins  with 
enough  money  for  2  bets  and  between  them  they  have 
enough  for  20  bets.  Thus,  the  probability  of  Gary’s  ruin  is 


q  2  = 


(1-5)2  —  (1-5)2 
1  -  (1.5)20 


=  0.9996. 


(iii)  For  £10  bets,  a  =  10  and  j  =  1;  and  the  probability  of 
Gary’s  ruin  is 


1.5  —  (1.5)10 
1  —  (1.5) 1 0 


0.9912. 


(iv)  Gary’s  best  chance  is  to  stake  all  his  money  in  a  single 
bet  of  £10. 
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2.4(i)  We  must  show  that  D}  =  j/(q  -  p)  satisfies 
Equation  (2.2).  Substituting  in  the  right-hand  side  of  (2.2) 
gives 


Pdj+  1 


+  qDj-i  +  l  = 


p(j  +  !)  [  q(j  -  i) 

q-p  q-p 


+  1 


j  +  p-q 

q-p 


+ 1, 


since  p  +  q  —  1, 


= - -  =  £>.-,  as  required. 

q-p  1 

The  general  solution  of  a  non-homogeneous  recurrence 
relation  is  obtained  by  adding  a  particular  solution  to  the 
general  solution  of  the  corresponding  homogeneous 
recurrence  relation  (which  is  given  for  this  case  in  Frame  4). 
So 

Dj  =  -i—  +  A  + 
q-p 

where  A  and  B  are  constants. 


(ii)  D0  =  0  gives  A  +  B  =  0,  so  A  =  —B;  and  then 
Da  =  0  gives 

q  -  p  \pj 


{q-p){\  -  (q/ p)a) 

Thus, 

D  =_i _ a  ( 1  -  (q/p)j\ 

J  q-p  q-p\i-(q/p)aJ 


2.5(i)  Substituting  Dj  =  —j2  in  the  right-hand  side  of 
Equation  (2.2)  gives 

pDj+1+qDj^  +  1  =  — ±0  +  l)2 -•£(;-  1)2  +  1 
=  —j2  =  Dj,  as  required. 

So  Dj  =  —j2  is  a  particular  solution  of  Equation  (2.2)  when 
P  =  q  =  $',  the  general  solution  is  obtained  by  adding  this  to 
the  general  solution  of  the  corresponding  homogeneous 
recurrence  relation  (given  in  Frame  4),  so 
Dj  =  -j2  +  A  +  Bj, 
where  A  and  B  are  constants. 

(ii)  £>0  =  0  gives  0  =  A,  and  Da  =  0  gives 
0  =  —a2  4-  Ba, 
so  B  =  a.  Hence, 

Dj=  -j2  +  aj  —  j(a  -j). 

2.6(i)  In  this  case  a  =  1000,  j  =  500  and  p  =  q  =  so, 
using  Equation  (2.5), 

£>5oo  =  500  x  (1000  -  500)  =  250000. 

(ii)  In  this  case  a  =  1001,  j  =  1  and  p  =  q  =  so 
D,  =  l  x  (1001  -  1)  =  1000. 

So,  even  though  the  player  with  only  £1  has  probability  \  of 
being  ruined  on  the  first  bet,  the  average  length  of  a  match  is 
1000  bets! 


Section  3 

3.1(i)  (a)  After  1  step,  the  possible  positions  are  —1  and  1. 

(b)  After  2  steps,  the  possible  positions  are  —2,  0  and  2. 

(c)  After  3  steps,  the  possible  positions  are  —3,  —1,  1 

and  3.  s 

(d)  After  4  steps,  the  possible  positions  are  -4,  -2,  0,  2 
and  4. 

(ii)  (a)  P(X1  =  -  1)  =  q  and  P(Xt  =  1)  =  p. 

(b)  P( X2  =  —  2)  =  P(two  steps  of  —  1)  =  q2, 

P(X 2  =  0)  =  P(one  step  of  —  1  and  one  step  of  + 1) 

=  2  pq, 

P(X2  =  2)  =  P(two  steps  of  +  1)  =  p2. 

(c)  P(X3  =  -3)  =  q3, 

P(X3  =  —  1)  =  P(two  steps  of  —  1  and  one  step  of  + 1, 
in  any  order) 

=  3<?2p- 

Similarly,  P(A’3  =  1)  =  3  p2q  and  P(X3  =  3)  =  p3. 

3.2(a)  When  n  is  odd,  the  possible  positions  of  the  particle 
after  n  steps  are  ±  1,  ±  3, . . . ,  +  n. 

(b)  When  n  is  even,  the  possible  positions  of  the  particle 
after  n  steps  are  0,  +2,  ±4,  ...,  +«. 

3.3(i)  For  n  =  10,  k  =  4  we  have  (n  +  k)/2  =  7  and 
(n  -  k)/2  =  3,  so 

P(Xl0  =  4)  =  (0.6)7(0.4)3  ~  0.215. 

(ii)  For  n  =  15,  k  =  3  we  have  («  +  k)/2  =  9  and 
( n  —  k)/2  =  6,  so 

P(X15  =  3)  =  (^(O.fim^6  *  0.207. 

(iii)  For  n  =  20,  k  =  —2  we  have  (n  +  k)/2  =  9  and 
( n  —  k)/2  =  11,  so 

/>(*,„  =  -2)  =  (29°)(0.6)W‘  =  0.071. 

3.4(i)  E(Zi)  =1  x  p  +  (—1)  x  q  =  p  -  q; 

E(Zf)=l2xp  +  (-l)2xq=l. 

So  V(Zj)  =  1  -(p-q)2 

=  (p  +  q)2  -(p-  q)2 

=  *pq- 

(ii)  Since  Xn  =  Zj  +  Zj  +  ...  +  Z„, 

E(Xn)  =  E(Z1)  +  E(Z2)  +  ...  +  E(Zn) 

=  n{p  —  q),  using  part  (i), 
and,  since  Z,,Z2, Z„  are  independent, 

V(Xn)  =  F(ZJ  +  V(Z2)  +  ...  +  V(Z„) 

=  4 npq,  using  part  (i). 

3-5(i)  g(p)  =  4 p  -  4 p2,  so 
g'(p)  =  4-8  p 

=  0  when  p  = 
and  g"{p)  =  -  8  <  0. 

So  g{p)  takes  its  maximum  value  when  p  =  and  this  value 
is  1. 

(ii)  For  a  simple  random  walk  with  p  +  q  =  1, 
v(Xn)  =  4 npq  =  4np(l  -  p). 

Thus,  for  each  n,  the  variance  of  the  position  is  a  maximum 
when  p  =  q  =  that  is  for  a  symmetric  random  walk. 
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3.6(i)  When  p  =  0.6, 

E(X„)  =  n( 0.6  -  0.4)  =  0.2n 
and 

K(AT„)  =  4  x  n  x  0.6  x  0.4  =  0.96n. 

(ii)  When  p  =  0.8, 

E(Xn)  =  n(0.8  -  0.2)  =  0.6n 
and 

V(X„)  =  0.64n. 

(iii)  When  p  =  0.9, 

E(X„)  =  n( 0.9  -  0.1)  =  0.8n 
and 

V(X„)  =  0.36  n. 

3.7(i)  When  p  =  0.5, 

E(X100)  =  ®  and  ^ioo)  = 

by  the  formulas  in  Question  3.4,  so 


P( 

-20  <  X100  <  20) 

~  P(—  1.95 

<  Z  <  1.95) 

(using  a 

continuity  correction) 

=  2<t>(  1 .95) 

-  1  =  0.9488  ^  0.95. 

(ii) 

When  p  = 

0.6, 

E(Xl00)  =  20 

and 

V(X100)  = 

96, 

so 

P( 

-20  <  X100  <  20) 

—  0.48. 

(iii) 

When  p  = 

0.9, 

E(Xl00)  =  80 

and 

V(X10o)  = 

36, 

so 

P( 

-20  <  AT100  <  20) 

^  0. 

(For  p  =  0.9,  notice  that  after  only  100  steps  the  probability 
of  being  within  20  units  of  the  origin  is  approximately  zero. 
This  confirms  that  for  a  transient  random  walk  such  as  this, 
the  particle  almost  certainly  drifts  away  from  the  origin  very 
rapidly.  Even  for  p  =  0.6,  the  probability  is  less  than 
whereas  for  the  symmetric  random  walk,  which  is  recurrent, 
this  probability  is  approximately  0.95.) 

3.8  With  probability  0.95,  approximately,  the  particle  will 
be  within  1.96  standard  deviations  of  the  mean. 

(i)  When  p  =  0.5, 

E(X  100)  =  0,  V(Xl00)  =  100, 

so  1.96  standard  deviations  is  19.6,  and  the  required  range  of 
positions  is  (the  even  numbers)  from  —20  to  +20  inclusive. 

(ii)  When  p  =  0.9, 

E(Xl00)  =  80,  V(X100)  =  36, 

and  1.96V/K(A'100)  is  11.76,  so  the  required  range  of 
positions  is  (the  even  numbers)  from  68  to  92  inclusive. 

3.9  When  p  =  q  = 

E(X„)  =  0,  V(X„)  =  n 

and  \.96jV{Xn)  =  1.96^.  So 

(i)  when  n  =  900,  the  required  range  of  positions  is  —  58  to 
+  58  (evens  only); 

(ii)  when  n  =  3600,  the  required  range  of  positions  is  —  118 
to  +118  (evens  only). 


3.10(i)  £(Z,)  =  2  x  p  +  (-l)  x  q 

=  2p-q  =  3p-l 

and 

£(Zf)  =  22  x  p  +  (  —  l)2  x  q 
=  4p  +  q  =  3p  +  1; 
so 

F(Z;)  =  3p  +  1  -  (3p  -  l)2  =  9p  -  9 p2  =  9p(l  -  p). 
(ii)  Since  Xn  —  Z1  +  Z2  +  . . .  +  Z„,  we  have 
E(Xn )  =  n(3p  —  1)  and  V(Xn)  =  9np(l  —  p). 


(iii)  We  shall  follow  the  method  used  in  the  text  to  find  the 
distribution  of  Xn  for  a  simple  random  walk.  If  x  is  the 
number  of  steps  of  size  2  in  the  first  n  steps  and  y  is  the 
number  of  steps  of  size  —  1,  then  x  +  y  —  n  and 
Xn  =  2x  —  y  =  2x  —  (n  —  x)  =  3x  —  n. 

Let  Wn  represent  the  number  of  steps  of  size  2  in  the  first  n 
steps.  Then  W„  ~  B(n,  p),  so 


But  P{Wn  =  x)  =  P(Xn  =  3x  —  «),  so 


P(Xn  =  3x-n)  = 


Now  writing  k  =  3x  —  n,  so  that  x  =  (n  +  k)/3  and 
n  —  x  =  (2 n  —  k)/3,  we  have 

n  )  (n  +  *)/3  (2H-J0/3 

(n  +  k)/3jP  Q 

This  holds  for  values  of  n  and  k  such  that  n  +  k  is  a  multiple 
of  3  and  —n<k<2n\  that  is,  for  k  =  —n,  —  n  +  3,  —  n  +  6, 
. . . ,  2n  —  6,  2n  —  3,  2 n.  Otherwise,  P(X„  =  k)  =  0. 


P(Xn  =  k)  = 


(iv)  Using  part  (ii)  with  p  =  $,  we  have 

(a)  E(X20)  =  0,  and 

(b)  V(X20)  =  40. 

(c)  For  n  =  20  and  k  =  0,  n  +  k  is  not  a  multiple  of  3,  so 
P(X20  =  0)  =  0. 

(d)  For  n  =  20  and  k  =  1,  (n  +  k)/3  =  7  and  (2 n  —  k)/3  =  13, 
so 


(v)  When  p  = 

E(Xy  80)=  -90 
so  by  Result  (3.8), 


and  V(Xieo)  =  225, 


+  90 


15 


JV(  0, 1). 


Hence,  using  a  continuity  correction,  we  have 


Pi-10  <  XI80  <  70)  -  P[  -69-5,-+9°  <  Z  <  69  5  +  90 


15 


15 


=  P(1.367  <  Z  <  10.633) 
=  0(10.633)  —  0(1.367) 

=  1  -0.9142 
~  0.086. 
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Section  4 


4.1(i)  There  are  three  ways  in  which  a  return  to  the  origin 
can  occur  after  three  steps:  the  particle  returns  to  the  origin 
for  the  first  time  after  one,  two  or  three  steps.  If  the  first 
return  occurs  after  one  step,  then  a  further  return  must  occur 
two  steps  later;  and  if  the  first  return  occurs  after  two  steps, 
then  a  further  return  must  occur  after  one  more  step.  Thus, 
the  three  events  have  probabilities  f1u2,f2ul  and  f3  =  f3u0 
(since  u0=  1),  and 

U3  =  /l«2  +/>!  +/3“0 

=  fou3  +  fi  «2  +  fz u  1  +  /s  w0  (since  f0  =  0). 

Similarly,  for  a  return  to  the  origin  to  occur  after  four  steps, 
the  first  return  must  occur  after  one,  two,  three  or  four  steps 
(with  further  returns  after  three,  two,  one  or  zero  more  steps, 
respectively);  so 

W4  ~  flU3  +  f2U2  +/3U 1  +  Awo 

=  foUA  +JlU3  +/2«  2  +./3W  1  +/4U0- 

(ii)  If  a  return  to  the  origin  occurs  after  n  steps,  then  the 
first  return  must  occur  after  one,  two,  ...,  or  n  steps  (with  a 
further  return  after  n  —  1,  n  —  2,  ...  or  zero  more  steps, 
respectively);  so 

Un  =  j\Un-\  +  f2Un-2  .+  •  •  •  +  fnU 0 
=  /oW«  +  flun-l  +  •••  +f„U0. 


4.2  The  term  in  s2m  in  Formula  (4.7)  is 

/•  „2m  (2M  2>(  2)  X  ...  x  (i  -  m  +  1),  „  ?s 

J2mS  - — - (-4  pqs2) 


(2 m  —  3)(2m  —  5)  x  . . .  x  3  x  1 


m!  2" 


(4  pqs2 


1 


(2m)! 


2m  -  1  m!2m2rn(2rn  -  2)  x  ...  x  4  x  2 

1  (2m)! 


(4  pqs1 


2m  -  1  m!m!  ^ S 

~(2m)pVs>' 

2m  —  \\m  J 


1  2m 


Hence, 

/2m  =  ^ - 7 1  I  Pmqm, 

2m  -  1  V  m  ' 

and  from  Formula  (4.4)  we  have 

,  1 

j2m  2m  -  1  U2"r 


4.3  For  a  simple  random  walk  with  p  =  q  = 
Formula  (4.8)  simplifies  to 


/2i 


1  /  2m 

2m  —  1  \  m 


(i)  We  require  the  value  of  /4,  so  m  =  2  and 
f*  =  =  0.125. 


(ii)  For  10  steps,  m  =  5  so 
/,o  =  ^(150)(i)1O“a027- 


(iii)  For  20  steps,  m  =  10  so 
/20  =  A^jo^  ^2°  —  0009. 


(The  binomial  coefficients  are  given  in  the  table  on  page  44 
of  Neave.) 


4.4(i)  The  value  of  /6  is  required  when  p  =  0.6  and  q  =  0.4. 
This  is  given  by  Formula  (4.8)  with  m  =  3,  so 

fe  =  i(^j(0.6)\0.4)3  ~  0.055. 

(ii)  The  value  of  /]2  is  required  when  p  =  0.3  and  q  =  0.7. 
This  is  given  by  Formula  (4.8)  with  m  =  6,  so 

/12  =  rr('62)(0.3)6(0.7)6  ~  0.007. 


Section  5 

5.1  When  Xn_1  =  a  —  1,  the  distribution  of  Zn  is  given  by 
P(Z„  =  0|Z„_1=a-l)  =  p, 

P(Z„  =  —  1 1  Xn_ ,  =  a  —  1)  =  q  =  1  —  p. 

This  is  illustrated  in  the  diagram  below. 

q 

P 


5.2  For  k  =  1,  that  is  for  X„  =  1,  the  only  possible  values 
of  A'„_1  are  1  and  2;  also,  Gary  cannot  win  the  nth  bet  and 
have  only  £1  afterwards,  so  he  must  lose  the  nth  bet;  and  the 
probability  that  he  does  so  is  q  whether  he  has  £1  or  £2  after 
n  —  1  bets.  So 

Pli  =<inXn-i  =  11*0  =./)  +  qP(Xn„y  =  2|*0  =j) 

=  qp{"sl)  +  qp(i,: 2_1)- 

Similarly,  for  k  =  a  —  1,  that  is,  for  Xn  =  a  —  1 :  Gary  cannot 
lose  the  nth  bet  and  have  £(a  —  1)  afterwards,  so  he  must  win 
the  nth  bet  (this  has  probability  p);  he  can  have  either 
£(a  —  1)  or  £(a  —  2)  after  n  —  1  bets.  So 

P{&-x=PP(  =a-2\X0=j) 

+  pP(Xn_l=a-l\X0=j) 

=  PPf.a-2  +  PPta-l 

5.3(i)  Substituting  for  7c,  in  Equation  (5.7)  using 
Equation  (5.6)  gives 
n2  =  qn2  +  qn3, 

P 

so  n3  =  -tc2. 

Q 

(ii)  Setting  k  =  3  in  Equation  (5.3)  gives 

n.3  =  pn2  +  qn 4; 

then  substituting  for  n2  using  part  (i)  leads  to 


P 

7t4  —  71  v 

q 

Repeating  this  process  leads  to 


5.4  When  p  =  q  =  p/q  =  1.  Then,  using  the  argument  in 
Solution  5.3,  we  obtain  nk  =  nk_t  =  . . .  =  7t2  =  nl ,  so  all  the 
nk  are  equal.  Since  the  sum  ny  +  n2  +  ...  +  na_1  is  1,  each 
probability  is  equal  to  1  /(a  —  1). 
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by  Question  5.4, 


5.5(i)  The  average  fortune  is 

a~ 1  l  1 

V  knk  =  -(a  —  1  )a  x  - 

*=i  2V  a  —  l 

1 

-  2°' 

(ii)  Writing  x  for  p/q,  we  have 

*  1  -  x  k_1 

nk  = - -x  . 

k  1-x"-1 

So  the  average  fortune  is 

k=  1  1  —  X  A=  1 


1  -  X 


1  -  X' 
a-  1 

where  S  =  £  kxk~K 

k=  1 


-xS, 


Now 

and 


so 


S  =  1  +  2x  +  3x2  +  ...  +  {a  -  \)xa~2 

xS=  x  +  2x2  +  ...  +  (a-  2)xa ~ 2  +  (a  -  l)xa~\ 

(1  —  x)S  =  1  +  x+  x2  +  . . .  +  xa~2  —  (a  —  l)xa_  1 

1  -x0-1 


1  -  x 


-  (a  -  l)x‘ 


and 


S  = 


1  -  x" 


(a  -  l)xa 


(1  -  x)2  1  -  x 

So  the  average  fortune  is 

_1 _ (a  -  l)xfl~! 

1  -  x  1  -xa_l  ’ 
where  x  =  p/q. 


5.6  If  T  is  the  time  spent  in  state  a  —  1,  then 
P{T  =  t)  =  p'~  lq. 

That  is,  T  has  a  geometric  distribution,  T  ~  G^q),  and  this 
has  mean  i/q. 
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